Strategy Soup The evidence

The Long Game Project · Methods · as at 23 October 2026

How do we use AI agents to make a simulation?

Not by asking a chatbot. Strategy Soup, the Long Game Project’s lab, turns each question into a game of the kind wargamers play, built on checked facts and rules we wrote. AI agents play the organisations, then the game is replayed 100 to 7,200 times. A report shows which results hold across the replays, and which choices change them. An expert, named on each report, signs the question first and the report last.

Strategy Soup is the Long Game Project’s lab. A report is not one answer from a chatbot. Each one is a game about a real decision, built on facts we have checked and rules we wrote. In most reports, AI agents play the organisations in a live game, and fixed rules then replay it many times to see which results hold. An expert, named on each report, owns the question at the start and the report at the end.

Why it matters. A chatbot answer comes from what the AI already believes, with no record of how it got there. Each of our reports is recorded at every step, so a reader can see where each number came from and who decided what. This page shows where AI agents do the work, where fixed rules take over and where an expert decides. It leaves out the rule tables and the instructions inside our engine, the software that runs each game.

Ask a chatbot Will our plan work? One answer. × from what the AI already believes × no record of how it got there × no check against the facts
A Strategy Soup report a signedquestion 313 claimsin 7 reports AI agents playa live game fixed rulesre-run it an expertsigns

Pre-game · Concept

Each game starts with a good question.

An expert writes the question in the words of the people who would have to make the call, and signs it before anything runs. The brief also fixes in advance what we expect to find, and what would stop us publishing.

Pre-game · Research

AI agents check the facts first.

AI research agents collect what each organisation has said and done in public, and a second AI pass checks every quote against its source. Of 313 claims checked across seven reports, 262 were supported, 44 partly, and the 7 that failed are never used. Client material is kept apart and never appears in a public report.

Pre-game · Roles

Each organisation gets an AI agent to play it.

The agent works from a written profile of how that organisation decides: its goal, how it wins, its red lines and its read of the others.

Game · Play

AI agents play a live game, then fixed rules replay it many times.

Each move has a set chance of working, fixed before play with its reason recorded. In the live game, each AI agent chooses its organisation’s moves, and the engine, not the AI, makes the random draw that decides whether a move works. Fixed rules, a written policy for each player, then replay the game with new draws, 100 to 7,200 times depending on the report. A report counts how often each result happened under the game’s rules. It is not a forecast of the real event.

Post-game · Report

An expert reviews and signs the report.

An AI agent drafts the report from the record. An expert edits and signs it. A number with no source stops publication.

Pre-game · Concept

Each game starts with a good question, signed before anything runs.

An expert, named on each report, writes the brief and signs it. It states the question in the words of the team that would have to make the call: the decision they would lose sleep over. It names the answer the game will test and the kind of game that fits. Format follows question, never the reverse.

Everything after this is checked against the brief. For a commissioned report, the client signs the brief instead.

A change to a signed brief is written under the signature, with its date, so the first version stays on file.

Pre-game · Concept

The brief also says what would stop us publishing.

The kill criteria are the results that would mean we do not publish. No run has ended there yet. Some briefs also seal the results we expect, before play. For report 008, six expected results were written down before the code that runs the game existed.

Of report 008’s six sealed results, three held, one held in part, and two broke, one of them on some runs only. The sealed list is dated and kept on file. A sealed result that breaks is a finding in its own right.

Pre-game · Brainstorm

AI agents argue against each draft question, and an expert keeps the one that survives.

The expert and AI agents draft more questions than we run. AI agents map the public debate on each one and argue against it. A question survives four tests. It is contested, argued both ways in public. It is timely, tied to a live news or budget cycle. It is useful to whoever would make the call. And it is honest: about how a situation moves and forks, never a forecast.

Report 005 started from five draft questions and kept one. For report 008, AI research agents read eleven published plans. Three claim some control over insiders, and only one names the threat the report tests and claims a control against it. That one became the subject. Both briefs record why the others lost.

This part of the work has the least written down: there is no fixed way to draft the candidates. The four tests are fixed, and so is the record of why each candidate lost.

Pre-game · Research

AI research agents gather the facts, and a second AI pass checks every quote.

Before play, AI research agents collect what each organisation in the game has said and done in public. Each claim records its publisher, its date, a quote word for word, the date it was retrieved and a verdict. The claims for one report make its evidence pack.

A second pass, also run by AI, checks each quote again. Does it exist, and does it support the claim?

Pre-game · Research

Of 313 claims checked, 7 failed, and those 7 are never cited.

Only a supported or partly supported claim can be cited. A partly supported claim is one its quote backs only in part. The packs cover reports 001, 002, 005, 006, 008, 009 and 010. Report 007 measures AI systems directly, so it has no pack.

Each pack is marked public or client. The engine refuses to use a client pack in a public report.

Pre-game · Kind of game

The question picks the kind of game.

If the question turns on players arguing their case, it becomes an argued game, known in wargaming as a matrix game: each player argues for a move, its rivals argue back, and a referee rules. If it turns on plans over time, with costs and chance, it becomes a mechanical turn-based game. If players compete or cooperate to get somewhere first, it becomes a race game, as in report 009. A perpetual game stays open and is played again as events arrive, as report 005 is. Report 007 measures AI systems directly, with no game.

AI agents help write each game: its code, its move menus, its table of chances and the policies that play the re-runs. Each chance must carry its reason, and an expert checks the ones that rest on design logic. A brief can also name an optional control run: the same game with the one thing under test switched off. If the control shows the same result, the result does not depend on that thing. Reports 005 and 008 name one. For report 005 the control gave the same endings, and the report says so.

Pre-game · Mechanics

A turn runs the same way every time, and the engine makes every random draw.

In a turn-based game, every player chooses a move at the same time. Each move has a chance, its stated probability, such as 40 in 100. The engine then makes a draw, a random number, for each move against its chance. Effects land at once, or as a delayed bill a few turns later, and the next turn starts. Each chance carries its reason: the evidence, a base rate, a real analogue or the design logic.

No AI player or referee can change a draw. Re-runs keep every chance and change only the draws. Runs with the same draws can compare two plans with nothing else changed.

Chance cards, built but not yet in a report, will show where each chance came from and how sure we are, and stop one piece of evidence counting twice. In an argued game, a referee rules on how likely each move is to work, with its reasons, and a draw settles it. The referee can be an expert, an AI agent or fixed rules.

Pre-game · Mechanics

The Domains board plays several contests at once, one layer for each.

A game can now run on a board with layers: sea and air, public opinion, markets, and the forums where votes are cast. Each layer has places. Each player’s assets sit at places. What each player holds at a place decides who controls it, who contests it and where it is open.

Layers act on each other. A blockade at sea can move opinion at home, and sanctions in a market can weigh on a vote.

An asset can be a ship, a factory, cash or political capital. A player names the assets a move uses, and a fixed rule fills any gap. An AI player will get a written account of the board before each move, drawn only from what its organisation can see.

Pre-game · Mechanics

A threat can deny a place without taking it, and no player sees the whole board.

Any asset can make a place risky or costly for its rivals: a shore battery over a sea lane, a patent suit over a market, sanctions over a vote. Control still goes by what each player holds, so one side can hold a strait that the other side denies.

Each player sees the board only as its organisation could. A hidden asset is seen only when another player’s sensor picks it up, even when both are in the same place.

Players decide once a turn, and their orders then run day by day inside it. Assets use up stocks such as fuel, munitions or cash, and refill them slowly. A game can also set an escalation line: enough clashes between two players in a short time ends the run, recorded as escalated to conflict.

Each of these parts is switched on game by game, so a game that does not use one runs exactly as it did before.

Pre-game · Mechanics

Players can talk before they move, and the record shows who kept their word.

Before the moves, players can make deals, promises and threats, set an ultimatum or leak what they were told. A deal both sides accept is carried out on the board. The record shows which words had something behind them: a threat with nothing behind it is recorded as a bluff, even when it worked.

Each AI player also keeps a private notebook of what it believes the others will do, so its guesses can be scored after the game.

When a report reads from the notebook that a promise was never meant, it marks that reading provisional until a test shows the notebook predicts kept and broken promises better than chance. Fixed-rule players can talk too, so the re-runs can include talks, and they can be set to play from their own view of the game, as AI players do.

Pre-game · Mechanics

The new parts are built and tested, and none is in a published report yet.

A new part must change nothing in the games already published. We replayed 9,304 recorded worlds, each one a complete game, including runs of the games behind reports 002, 009, 010 and 011. With the new parts added, every one came out the same. The board’s opinion layer also matches report 011’s public-opinion measure exactly, at all 81,000 time steps.

An outside wargaming expert reviewed the board, and we agreed 11 changes. The eight that change the engine are built. The other three are a second starting point for report 011, a rule for how a report reads the notebook, and a player aid we will write. Report 011’s second version will be the first report to use the board.

The board’s market layer matches our earlier market-share game exactly, at every time step. 2,690 automated tests pass, as at 22 October 2026.

Pre-game · Roles

AI agents play the organisations, each from a written profile.

Each player gets a profile of how one organisation decides: its goal, how it wins, its red lines and its read of everyone else. A profile is built from the organisation’s public record. Where a report models how an organisation decides, a questionnaire scores that first. Its results are on the evidence page.

AI agents play the profiles in live games. Fixed rules play the many re-runs: a written policy for each player that picks its move from the state of the game, with no AI in the loop, so only the draws change between runs. An invented player is named by what it does, such as a regulator. A real organisation keeps its name.

An AI player works from its profile and the state of the game. On the new board, a player sees only what its organisation could know. Fixed-rule players can already play from that view, and AI players will get it before each move. The questionnaire is answered by an AI reading a written brief about the organisation. Asked again, it gives much the same scores: its mean ICC, the share of score variation that is real difference between organisations, is 0.84.

Pre-game · Roles

We test whether an AI player plays its profile, and we publish the misses.

Blind AI readers saw transcripts of AI play with every name stripped out. They named the right player, one of four, in 19 of 24 transcripts, where guessing would get about 6. Offered the same real business deal, one player signed in 9 of 9 runs, and another refused in 10 of 10.

Two marks against. Every AI player, with a profile or without, states odds 14 to 28 percentage points higher than the share of its moves that then work. That bias does not set the published counts: in the re-runs behind them, each chance is fixed in the game’s table before play, with its reason, and no player’s stated odds enter it. And all the player tests used one small AI system, Claude Haiku 4.5, in one kind of business scenario. None of those results carries over to other fields on its own. The full evidence, and what broke.

Pre-game · Moves

Each player picks from a menu, and every move shows its chance and its reason.

In a turn-based game, each player picks one move a turn from a fixed menu. Each move names what it needs, what it costs now and later, and when it lands. Some moves are certain. Repeating a move costs more. An AI player gives its reason in its own words, and the reason is recorded with the move.

Each chance shows its reason where the reader meets it. Where no evidence, base rate or real analogue exists, the reason is the design logic, labelled so a reader can tell it from evidence. Every move is shown on each report’s Deep page.

In an argued game, a player states its move in one sentence with its odds, and every rival argues against it in writing. No player grades its own move. When fixed rules play an argued game, as in reports 005, 006 and 008, the moves are set before play and a rule-based referee rules on each.

Game · Play

Every claim, ruling and draw is recorded, in a log that only ever grows.

If an expert overrules the referee, the first ruling stays in the log, and the override sits beside it with the expert’s reason.

A game played by fixed rules comes out the same when re-run with the same draws. An AI ruling cannot be repeated that way, so a game with AI players or an AI referee is replayed from its log, which keeps every ruling as it was made.

Before a game’s numbers are published, our export replays each world and stops on any mismatch. That check covers the turn-based games. It does not yet cover the argued games of reports 005, 006 and 008, whose numbers were reproduced from their run records.

Game · Play

One live game, then many re-runs.

In reports 002, 009, 010 and 011, AI agents played a live game first, choosing their own moves. Fixed rules then replayed the same game many times, from 100 to 7,200 runs, changing only the draws, to see which results hold.

In the live games of reports 010 and 011, the AI players ran on Claude Opus 5.5.

The live game and the re-runs are two plays of the same game, with the same rules and the same starting position. In the live game, AI agents choose each move and record why. In the re-runs, a written policy for each player chooses, and only the draws change from run to run. The counts in a report come from the re-runs. The live game supplies the moves and reasons a report quotes.

A fork changes one thing and runs on: different draws, one outside event removed, or one decision changed at one turn. After a fork, fixed rules play on.

The re-runs show which results change with chance. They do not give the odds of a real event, and we do not claim them. The table after the journey shows who played each report and how many times it ran.

Post-game · Analysis

We read results as counts, and compare plans on the same draws.

A result reads as a count: how many of every 100 re-runs it happened in. To compare two plans, each pair of worlds shares its draws, so the plan is the only difference. In report 002, the baseline plan ended on a higher final position in 57 of 100 pairs, the early pivot in 39, and 4 tied.

Then an expert reads the whole record against the signed question, and picks the five to eight findings the argument rests on. This is where the kill criteria would stop a report.

Paired runs on the same draws cut out the noise between two plans. With 96 pairs that did not tie, a 57 to 39 split leans towards the baseline, but a split that uneven would still turn up about 8 times in 100 by chance alone. Fifteen runs per setting, as in reports 005, 006 and 008, are enough to see which results change with chance, and too few to state a precise rate. Each report’s Deep page shows every move, so a reader can check the findings against the whole game. For report 008, a fresh AI reader with no part in the draft also checked the findings against the record before they were signed.

Post-game · Analysis

Report 005 stays open, and its claims will be checked against what the US Treasury says.

Report 005 plays out three possible US Treasury plans for 4 November 2026. It is the first of our perpetual games: reports that stay open and are played again as events arrive. Its claims sit on a ledger.

On 4 November 2026 the US Treasury is due to state the reason for its plan. Each claim on the ledger will then be marked against that stated reason.

Post-game · Report

An AI agent drafts the report, and an expert reviews, edits and signs it.

The draft comes from the record and the evidence pack. The expert reads it against the signed question. The engine cannot pick the argument a report makes. Where the author goes beyond what the game showed, the report says so in its own text.

Each report is written at three reading levels, Easy, Normal and Deep, with the same numbers in each. Before the results, each report with players sets out the lay-down: what each side has, what each can see, how each wins, and which starting values move the outcome. Two checks stop the build: a number with no citation, and a number that does not match the record.

ConceptPre-game
The signed brief The questionin the decision-maker’s words The answer it testswhat we expect to see The kind of gamechosen to fit the question What stops publicationthe kill criteria Signedbefore anything runs The findings The report read against STOP Not published if a kill criterion is metNONE SO FAR Report 008: six expected results,sealed before play
From many draft questions to one AI agents map the debateand argue against each draft draft questions ContestedTimelyUsefulHonest an expertkeeps one 005: 5 drafts → 1008: 11 plans → 1
Sources to checked claims AI researchagents 313 claims One claim, report 002 PublisherIntercom Date26 Aug 2026 Quote“$0.99 per outcome” Retrieved26 Aug 2026 Verdictsupported a second AI pass checks every quote 262 supported 44 partly supportedcan be cited 7 not supportednever cited Kept apart Public pack Client pack
Picking the kind of game The question Argued gamea matrix game: argued, then ruled005006008 Mechanical turn-basedmoves, costs and chance, each turn002010011 Competitive or cooperative racewho gets there first, alone or together009 Perpetual gamestays open, replayed as events arrive005 Measurement studythe AI systems themselves007 Optional control runs = ONE THING SWITCHED OFFREPORTS 005, 008
One turn, run the same way every time Talksdeals and promises Moves chosenall players at once The engine drawsdraw: works Effects landnow, or as a delayed bill The board updateswho has what, where The logrecords every step dashed: built, not yet in a published report
One board, in layersThreat, sight, time and supply SEA AND AIR PUBLIC OPINION MARKETS FORUMS AND VOTES a strait what each player holdsdecides who controls it a blockade at seamoves opinion sanctions weighon a vote held by one player,denied by another hidden: seen onlyby another’s sensor ONE TURN, DAY BY DAY an order arrives on day 5 STOCKS fuel, munitions, cash:used up, refilled slowly TOO MANY CLASHES the run ends held by one playerheld by anothercontestedopen
Talks, then the moves an incumbenta challengera regulator offers a deal promises threatens leaks the threat an ultimatum: public, with a deadline The record the incumbent’s offeraccepted, done on the board the challenger’s promisekept the incumbent’s threata bluff: nothing behind it Example talks, not from a report each player’s private notebook Built and checked, as at 22 Oct 2026 9,304 recorded worlds, replayed without the new parts = with them 0 differ Report 011’s opinion measure, on the board the old measure = the board’s layer at all 81,000 steps An outside wargaming expert: 11 changes agreed 8 change the engine, and are built 3 more: a second start for 011, the notebook rule, a player aid THE FIRST REPORT TO USE THE BOARD Report 011, second version 2,690 TESTS PASS
AI agents play the organisations The refereeexpert, AI or fixed rules the engine ownsevery draw a regulator an incumbent a challenger a buyer profile: goal, how it wins, red lines, its read of the otherslive gamere-runs: a written policy for each player Does an AI player play its profile? Blind AI readers named the right player guessing: about 6 19 of 24 Offered the same deal, who signed 9 of 9 signed 0 of 10 signed Stated odds against what works WHAT WORKS WHAT THEY STATE 14 to 28 points too high A mark against. Not in the counts: the re-runs use each move’s set chance.
One move card, and a turn of choices The moveexample: “cut the price” What it needscash in hand What it costsmargin now, a price war later Its stated chance40 in 100, from a base rate When it landsnext turn Example move, not from a report All players choose at once revealed together worksfailsdelayed bill each move and its reason is recorded
A log that only ever grows claimobjectionrulingdrawclaim… override + reason Played by fixed rules = same draws, same result Played by AI agents ruling A ruling B≠ asked again ruling A= log replay One live game, then many re-runs a fork different drawsone event removedone choice changed 100 to 7,200 re-runs: fixed rules, only the draws change the live game: AI agents choose WHERE RUNSENDED→ ANALYSIS
Report 002: 100 pairs, each on the same draws baseline higher: 57 early pivot higher: 39 tied: 4 Top square: the baseline plan. Bottom: the early pivot. The record an expert reads itagainst the question kill criteria 5 to 8 findings
Report 005: a report that stays open The claim ledger4 NOV ???? Sep 2026Oct events arrive, and the game is played again 4 Nov 2026the US Treasury is dueto state its reason
From a draft to a signed report the record the evidence pack an AI agentdrafts CITED? MATCHES? SIGNED Easy57 of 100 Normal57 of 100 Deep57 of 100 Report written atthree reading levels STOPbuild stops
On file so far The signed brief Why this question won The evidence pack The game and its control Rules, chances and the board Player profiles Move menus The game record The findings The report, three depths

The biggest limit. Our AI players overrate their own chances of success by 14 to 28 percentage points, and we are still fixing it. It does not change the chances the replays use: each move keeps its set chance, not the AI’s guess.

This page shows what our rules, profiles and engine do, not how they are built.

Who played each report

An expert ruling on every move, an AI referee and fixed rules are different things, and we will not blur them.

ReportMoves chosen byRuled byRuns
001AI agentsAn expertOnce, with no chance draws
002AI agents, then fixed rulesAn AI referee, then fixed rules1 live game, then 100 paired re-runs
005Fixed rulesFixed rules45, 15 per Treasury plan
006Fixed rulesFixed rules45, 15 per standard of proof
007No moves: AI systems measuredFive blind AI judges17 AI systems, then 8 on 30 scenarios
008Fixed rulesFixed rules300, 15 per route under 5 sets of rules
009Fixed rules, and AI agents in one live gameThe rules and chance200 paired worlds per setting, plus the live game
010Fixed rules, and AI agents in 8 live worldsThe rules and chance300 for each of 3 forger setups, plus 8 live worlds, each played twice
011AI agents in one live world, then fixed rulesThe rules and chance1 live world, then 7,200 re-runs over 24 settings, plus 900 control runs

Reports 003 and 004 are planned and have not run yet.

Our hard rules

Rule 1 of 5Every number cites its source

Every claim with a number in it cites the evidence pack, or the public source it came from. A linter, a script that checks the prose, enforces this, and the build fails without it.

Rule 2 of 5Every run says what it is

A demo is labelled as a demo. A machine-played run is labelled as one. A real client engagement is named, with the client’s consent, or it is not mentioned at all.

Rule 3 of 5Client material stays private

Client-room material never appears in a public report. The engine keeps a wall between public evidence packs and client evidence packs. Crossing it is a hard error in the code.

Rule 4 of 5Who played, stated in one place

This page states how much of each report a machine played, in the table above. The reports do not repeat it.

Rule 5 of 5How many runs, for every report

The same table gives how many times each report ran. Re-runs show which results change with chance. They do not give the odds of a real event, and we do not claim them.

This page shows what our rules, profiles and engine do, not how they are built. A reviewer can see the rules and player instructions left off it on request.

Our AI agents run on Claude, made by Anthropic. Reports 007, 008 and 009 involve Anthropic directly, so read them with that in mind.

Who does what, and the check each part must pass

Each record can be checked against the one before it, so a report traces back to its signed question.

Pre-game

Concept
Who does the work

An expert, or the client for a commissioned report

Hands on

The signed brief and the kill criteria

Its check

Nothing runs without a signed brief. Changes are written under the signature.

Brainstorm
Who does the work

AI agents argue against each draft. An expert picks

Hands on

The record of why this question won

Its check

Contested, timely, useful and honest.

Research
Who does the work

AI research agents, then a second AI pass

Hands on

The evidence pack

Its check

Each quote checked again against its source. Only supported or partly supported claims can be cited.

Kind of game
Who does the work

An expert chooses. AI agents help write the code, menus and chances

Hands on

The game, and its control run where the brief names one

Its check

Format follows question.

Mechanics
Who does the work

AI agents draft each chance with its reason, and an expert checks those resting on design logic. The engine runs the turns and keeps the board

Hands on

The rules, the chances and the board, each with its reason

Its check

Automated tests. A new part must leave every recorded world unchanged.

Roles
Who does the work

AI agents in live games, written policies in re-runs

Hands on

A profile for each player

Its check

The player studies, published with their misses.

Moves
Who does the work

AI agents or fixed rules choose. The engine draws

Hands on

A menu of moves for each player

Its check

Every chosen number has a reason. Design logic is labelled as design logic.

Game

Play
Who does the work

The engine records. AI agents and fixed rules play

Hands on

The game record

Its check

Every claim, objection, ruling and draw is recorded. Overrides sit beside the ruling. A game played by rules comes out the same when re-run with the same draws. A game with AI players or an AI referee is replayed from its log.

Post-game

Analysis
Who does the work

The engine counts. An expert reads and picks

Hands on

Five to eight findings

Its check

Read against the signed question. The kill criteria apply here.

Report
Who does the work

An AI agent drafts. An expert reviews and signs

Hands on

The report, at three reading levels

Its check

A number with no citation, or one that does not match the record, fails the build.

What this page leaves out

This page leaves out the rule tables, the instructions the AI players and referees get, and the settings inside each game. Each report publishes its own evidence and, on its Deep page, every move.