Strategy Soup What we tested

The Long Game Project · Evidence · Study 2 of 5 · TLDR

P2 Contested forks: do personas change what an agent picks, not just how it writes?

P2 Contested forks

One genuine fork, two defensible lanes

An agent with a persona

Reaches the fork.

→
Sign

A binding three-year exclusive.

or
Take the pilot

12 months, reviewable.

The test reads which lane it picks, not how it writes.

The confirmatory fork, F2

Signed the exclusiveTook the pilot merchant fortress

one run, lane read from the markerone run with no choice marker, not scored

100 pointsapartNo overlap
v2 instrument

FAILED.

→
v3 instrument

Rebuilt.

→
The confirmatory fork

PASSES.

Corrected 2026-09-18

F1

Not read. Its baseline fails the liveness gate.

F3

Lands on the pass mark.

what the test showedfailed, or not read

Question

do personas change what an agent picks at a genuine fork, not just how it writes?

Rough answer

yes. The target pair split by 100 points, 9 of 9 against 0 of 10, no overlap.

Result

v2 instrument FAILED. v3 instrument rebuilt. The confirmatory fork PASSES. Corrected 2026-09-18: F1 is not read, because its baseline fails the liveness gate, and F3 lands on the pass mark.

Method · Validation · Study 2 of 5

P2 Contested forks: do personas change what an agent picks, not just how it writes?

Scope caveat, recorded 2026-08-17

This result comes from hand-written persona prose. The product ships the 54-question diagnostic pipeline instead, so the result stands as a result about our prose. The bridge study later reran this fork with questionnaire-built twins. It failed with a gap of +0pp, zero percentage points between the two characters. We fixed the renderer on 2026-08-18 and the retest passed at +80pp. Detail: the validation page.

Study
Protocol 2 of the persona fidelity programme
Question
do personas change what an agent picks at a genuine fork, not just how it writes?
Rough answer
yes. The target pair split by 100 points, 9 of 9 against 0 of 10, no overlap.
Run dates
v2 instrument 2026-08-11, v3 baseline and arms 2026-08-12 and 2026-08-13
Report date
2026-08-14
Result
v2 instrument FAILED. v3 instrument rebuilt. The confirmatory fork PASSES. Corrected 2026-09-18: F1 is not read, because its baseline fails the liveness gate, and F3 lands on the pass mark.
Design and orchestration
Claude Opus 5
Editorial review
Claude Fable 5
Pre-registration
the study protocols, dated 10 August 2026
Data
the run data, in our private repository
Access
this repository is private. Pre-registration documents and run data are available on request.
Two corrections, recorded 2026-09-18

All three forks reached in 40 of 40 runs.

The scorer's own output: 39 of 40

all three forks reached and scoredF2 a sign, no choice marker, never counteda line we took back

9 of 9already scored on 9No findingchanges

Scored from the claim line, not the first tag anywhere

F1 baseline blitzgeneric, 10 runsfloor 20%1 of 10 F2 gap9 of 9 against 0 of 10100 F3 gappoints, p = 0.170pass mark 4040

what the test showeda gate the fork failsthe first-tag count, before the correctiona mark we wrote down before the run

F1

Fails liveness. Reported, not read.

F3

From 50 to 40 points. Exactly the pass mark.

Two corrections, recorded 2026-09-18

One. Section 6 said all three forks were reached in 40 of 40 runs. The scorer's own output was 39. One merchant run gave its F2 answer, a sign, without the choice marker, so it was never counted. The 9 of 9 and the p-value were already scored on 9, so no finding changes.

Two. The scorer read the first choice marker anywhere in a transcript. The method section says the marker is written into the claim, and the pre-registered brief says a fork claim must begin with it. In five of the fifty v3 runs an actor wrote one lane in its reasoning and tagged the other on its claim. Every table below is now scored from the claim line. The confirmatory result, F2, does not move: 9 of 9 against 0 of 10. Two things do move. The generic baseline takes the blitz lane at F1 in 1 of 10 runs, not 2, so F1 fails its liveness gate and its discrimination result is reported but not read. The F3 gap falls from 50 to 40 points, exactly the pass mark, with Fisher p = 0.170. The report of 14 August called F1 a pass at the boundary and F3 a marginal pass. Neither description survives. The first-tag numbers reproduce with score-forks.py --first-tag.

Summary
P1

A reader can recover a persona from a transcript.

→
P2, the harder claim

Does the persona change the branch it takes?

A real fork

Two defensible branches.

Recorded in advance

Which branch each persona should take.

Read straight out

The branch, from the transcript.

what a test showed

The first instrument, v2. F2 cooperate share.

generic30% originalflipped against its prediction62% gambler40% fortress60%

a failed instrument

The pair it targeted, original and fortress, did not separate.

The rebuilt instrument, v3. The confirmatory fork, F2.

Signed the exclusiveTook the pilot merchant fortress

one run, lane read from the markerone run with no choice marker, not scored

100 pointsno overlapp = 1.08e-05about one chance in 90,000

The two secondary forks, after the correction of 18 September

F1 baseline blitzgenericliveness floor 20%10% F3 gappoints, p = 0.170pass mark 4040

what the test showeda gate the fork failsa mark we wrote down before the run

F1

Not read.

F3

Sits on the mark.

Summary

P1 showed that a reader can recover a persona from a transcript. A fair sceptic can still say the persona only changed the prose. P2 tests the harder claim. It gives agents a real business fork with two defensible branches, records in advance which branch each persona should take, then reads the branch straight out of the transcript.

The first instrument failed. Its confirmatory fork, the one fork whose result counts, did not separate the pair it targeted. The original persona also flipped against its own prediction. We rebuilt the instrument on three design rules and re-ran the test.

The rebuilt instrument passes on the one fork that counts. On the confirmatory fork the merchant persona signed a binding exclusive deal in 9 of 9 reached forks. The fortress persona refused it in 10 of 10 and took the reversible pilot. The gap is 100 points with no overlap. A Fisher exact test, the standard test for a difference between two small groups, gives a two-sided p = 1.08e-05, about one chance in 90,000.

The two secondary forks read worse after the scoring correction of 18 September. F1 is not read, because its generic baseline fails the liveness gate. F3 sits on the 40-point pass mark with p = 0.170.

1. Question
The choice

Which lane the agent takes.

or
The wording around it

Only how it writes.

What a choice test needs

1. A genuine fork

Both branches defensible to a competent agent.

2. The prediction first

Written before the run.

3. No model reads it

The outcome does not depend on a model's judgement.

1. Question

Does the persona change the choice, or only the wording around it?

A choice test needs three things. The fork must be genuine, so both branches are defensible to a competent agent. The prediction must be written before the run. Reading the outcome must not depend on a model's judgement.

2. Method

The claim line

“F2: sign”

The choice marker the agent writes into its claim.

→
A simple parser

Reads the marker on the claim line.

→
Lane: sign

No model in the scoring path.

Costs nothing

Anyone can reproduce it.

Until 18 September 2026

The first tag anywhere in the transcript.

Predictions written down before any run

Baseline

10 generic runs.

→
Gates read

Before any arm.

→
4 personas

10 runs each.

→
The fork

Binding deal or reversible pilot.

→
merchant 9 of 9

fortress 0 of 10.

Haiku 4.5 agents · a Haiku 4.5 arbiter · through the local shim

what the test showed

P2 study design: baseline gates first, then persona arms into a genuine fork

F2, the named target pair

signdid not sign merchant90 fortress010
Fisher exacttwo-sidedp = 1.08e-05on the lane share

runs, lane read from the marker

2. Method

Fork mechanics. Each fork presents two named lanes. An agent commits by writing a choice marker into its claim, for example “F2: sign”. A simple parser reads the marker on the claim line, so no model sits in the scoring path. Scoring a run costs nothing and anyone can reproduce it. Until 18 September 2026 the parser read the first tag anywhere in the transcript. The correction note at the top says what that moved.

Models and scale. Haiku 4.5 agents and a Haiku 4.5 arbiter, the engine's referee, through the local shim, a proxy that routes the model calls.

Analysis. A Fisher exact test, two-sided, on the lane share of the named target pair.

2. Method

batchrunsconditionscalls
v2 instrument364 (evangelist arm never ran)-
v3 baseline10v0-generic only480
v3 persona arms404 personas at 10 each1,489
3. The v2 instrument and why it failed

v2, F2, the pre-registered confirmatory fork. Cooperate share.

generic30% originalflipped against its prediction62% gambler40% fortress60%

a failed instrument

At this sample size the spread sits inside noise.

v2, F1, descriptive only. Blitz share.

gambler9 of 9100% original3 of 933% generic3 of 1030% fortress1 of 714%

a descriptive split, not the pre-registered test

p = 0.00087gambler against fortressp = 0.0031gambler against generic
F1

Descriptive. It split wide.

→
A hypothesis

Where it stays.

Promoted to a result after seeing the data.

our judgement, a rule we hold to

v2, F3: runs that reached the fork, per arm

1 to 4 0510
An engine fault

The personas played no part. Section 8.

a failed instrument

F2 lacked equipoise.

The first diagnosis. The data refutes it.

baseline splitold equipoise gatediscriminated F27030passedno F13070would have failedyes, p = 0.00087

the more aggressive lane, share of generic runsa diagnosis we corrected

Baseline balance was never the binding constraint.

Fault 1: one axis, how aggressive the actor is

originalfortress
cautiousaggressive
F1

Asked it.

F2

Asked it again. Added little.

Both of the pair F2 targeted sit at the cautious end.

Fault 2: the v2 prediction table

F1F2F3 generic··· original··· gambler··· fortress···

a prediction

No fork owned a job

Every persona on every fork. No fork accountable for one separation.

3. The v2 instrument and why it failed

F2 was the pre-registered confirmatory fork, and it failed. Cooperate share came out at generic 30%, original 62%, gambler 40% and fortress 60%. The original persona flipped against its own prediction. At this sample size the spread across archetypes sits inside noise.

F1 was descriptive only, and it split wide.

Fisher exact two-sided: gambler against fortress p = 0.00087, gambler against generic p = 0.0031.

F1 was not the pre-registered test, so it stays a hypothesis. Promoting a descriptive finding after seeing the data is the exact move this programme exists to prevent.

F3 was reached in only 1 to 4 runs per arm. The cause is an engine fault, and the personas played no part in it. See section 8.

Diagnosis, including a wrong first answer

The first diagnosis was that F2 lacked equipoise, meaning the two lanes did not each draw a fair share at baseline, because it split 70/30. The data refutes that. F2 split 70/30 and passed the old equipoise gate, then failed to discriminate. F1 split 70/30 the other way, passed the same gate too, and still discriminated at p = 0.00087. Baseline balance was never the binding constraint. We corrected the wrong diagnosis before we built any repair.

Two real faults caused the failure:

  1. F2 sat on F1's axis. Both forks asked how aggressive the actor is. So F2 re-measured F1 and added little. It also could not split the pair it targeted, because the original and the fortress both sit at the cautious end of that single axis.
  1. No fork owned a job. The prediction table covered every persona on every fork. So no fork was accountable for a specific separation, and a fork that separated nobody could still half-pass.

3. The v2 instrument and why it failed

conditionblitz share
gambler9 of 9, 100%
original3 of 9, 33%
generic3 of 10, 30%
fortress1 of 7, 14%
4. The v3 rebuild

Each fork names the pair it exists to separate. Open one.

The fork is judged on that pair alone.

Three forks, three separate axes

F1 aggression
consolidateblitz
F2 commitment shape
stagesign
F3 stance
undercutcooperate

No fork re-measures another.

F2, built for the confusable pair

Sign

A binding three-year exclusive. Fixes list pricing.

or
The pilot

12 months, reviewable. About 15% more later.

Both lanes carry quantified costs.

v2: measured at the same time

Baseline and persona arms

Together, to save wall-clock time.

→
The fault shows

After every persona token is spent.

v3: the order gate

Baseline first

10 generic runs.

→
Both gates read

Liveness, discrimination.

→
Then the arms

Persona runs start.

The tick budget, turns in a run

4 under v2
6 under v3
Before

Fork timing came from when an inject arrived.

Now

Fixed turn numbers.

the two turns the rebuild added

Pre-registered materials

The persona file, version 2

6af05669...

The fork scenario, version 3

01b9dada...

→
commit 1fe48a3

Both pinned to it.

4. The v3 rebuild

Three design rules, fixed before any v3 run:

  1. Every fork names the pair it exists to separate. The fork is judged on that pair alone.
  1. Forks sit on separate axes, so no fork re-measures another. F1 is aggression. F2 is commitment shape. F3 is stance.
  1. F2 is built for the confusable pair. It asks whether the actor signs a binding three-year exclusive that fixes list pricing, or takes a 12-month reviewable pilot that costs about 15% more later. Both lanes carry quantified costs.

The order rule stops wasted spend. The v2 batch measured equipoise at the same time as the persona arms to save wall-clock time. So a faulty instrument only showed after we had spent every persona token.

Two mechanical fixes came with the rebuild. The tick budget, the number of turns in a run, went from 4 to 6. Fork timing now comes from fixed turn numbers, and no longer from when an inject, a scripted event, arrives.

Pre-registered materials. the persona file, version 2, at 6af05669... and the fork scenario, version 3, at 01b9dada..., both pinned to commit 1fe48a3.

4. The v3 rebuild

The rebuild replaced the gate set:

gaterule
livenesseach lane draws 20% to 80% of generic-baseline runs
discriminationthe named target pair separates by 40 points or more. This is the pass mark
orderthe baseline runs first, and both gates are read before any persona arm starts
5. v3 baseline result

The v3 baseline: the generic agent, no persona

one generic run, no persona

10runs480callsabout 6hours

Did each run reach each fork?

reacheda range, not per run

F3 under v2runs out of 101 to 4

F1 on the claim line, run by run

blitzconsolidater9: blitz in its reasoning, the first tag we counted

Blitz lane1 of 10floor 20%80%10%

a gate the fork failsthe liveness gate we wrote down

5. v3 baseline result

10 runs, 480 calls, about 6 hours.

Every fork was reached in every run. Under v2, F3 landed in only 1 to 4 runs out of 10. The tick-budget fix is the clearest single win in the rebuild.

F1 fails liveness. The blitz lane drew 1 of 10 baseline runs, under the 20% floor. The first-tag scorer counted 2 of 10, because run r9 wrote [F1: blitz] in its reasoning and then consolidated on its claim, and the 14 August report recorded a pass at the boundary on that count. Under the protocol a fork with a dead lane is not read for discrimination, so the F1 row in section 6 is reported and not read.

5. v3 baseline result

forkbaseline splitreachedliveness
F1 aggressionconsolidate 90%, blitz 10%10 of 10FAIL, below the 20% floor
F2 commitment, confirmatorystage 50%, sign 50%10 of 10PASS
F3 stanceundercut 50%, cooperate 50%10 of 10PASS
6. v3 persona arms: the confirmatory gate passes

40 runs, four personas at 10 each

all three forks reached and scoredF2 a sign with no choice marker, not scored

39 of 40all three forks reached

F2: share of runs that chose the binding lane, sign

merchant100% gambler100% generic50% evangelist30% fortress0%

the pre-registered pair: +100 points, no overlap, p = 1.08e-05the other conditions

F2 sign share by condition, confirmatory pair highlighted

6. v3 persona arms: the confirmatory gate passes

40 runs, four conditions at 10 repeats each. All three forks were reached in 39 of 40 runs. In one merchant run the F2 claim was a sign but carried no choice marker, so it was not scored.

The persona repair worked. The pair that P1, P3 and P4 all confused now separates without overlap, on a fork built for that job.

6. v3 persona arms: the confirmatory gate passes

forktarget pairgapFisher two-sidedgate
F1 aggressiongambler 100% vs fortress 0% blitz+100pp1.08e-05not read, baseline liveness FAIL
F2 commitment, CONFIRMATORYmerchant 100% vs fortress 0% sign+100pp1.08e-05PASS
F3 stanceevangelist 80% vs merchant 40% cooperate+40pp0.170on the pass mark

Full lane shares:

conditionF1 blitzF2 signF3 cooperate
v0-generic10%50%50%
v1b-merchant90%100%40%
v2-gambler100%100%20%
v3b-fortress0%0%0%
v4-evangelist40%30%80%
7. Four caveats on the result

The full prediction table, written before the runs

predicted lane takena miss

10 of 12predictions right0misses on a gate

A reading made after the fact

merchant: blitz

Discounted bundles to at-risk accounts.

→
Defends revenue on price

What the merchant brief says.

fortress: undercut

Declines a public standard.

→
Refuses an unreviewed commitment

What the fortress brief says.

our reading, a hypothesis until a test confirms it

Persona against persona

The pre-registered test. Passed on F2.

Persona against generic, n=10

Did not pass. No claim rests on it.

Fisher two-sided, against the generic baseline

merchantF2p = 0.033 fortressF2p = 0.033 evangelistF3p = 0.350 gamblerF1p = 1.19e-04, not read

what the test showedunderpowered

F3 stance, cooperate share

evangelist80% merchant40%
The gappointspass mark 4040

what the test showedthe pass mark we wrote down

p = 0.170FisherDirectionalat best
F1 baseline blitz1 of 10floor 20%10%

So this is not read

gambler100% fortress0%

a gate the fork failsthe floor we wrote downreported, not read

7. Four caveats on the result

Two prediction misses. We predicted the merchant would consolidate on F1, and it blitzed in 9 of 10. We predicted the fortress would cooperate on F3, and it undercut in 10 of 10. Neither miss touches a gate, because each fork is judged on its named pair alone. The score on the full prediction table is 10 of 12.

A reading made after the fact says both personas stayed self-consistent and the predictions misread the briefs. Blitz means discounted bundles to at-risk accounts, which defends existing revenue on price, and the merchant brief says it defends revenue that way. Declining a public standard refuses an unreviewed commitment, and the fortress brief says it refuses those. Treat that reading as a hypothesis until a test confirms it.

The secondary comparison is underpowered. Measured against the generic baseline rather than against the paired persona, the merchant and the fortress each sit at p = 0.033 on F2, Fisher two-sided from the lane-share table above. The evangelist sits at p = 0.350 on F3. The gambler reaches p = 1.19e-04 on F1, and that fork is not read. Pair separation was the pre-registered test and it passed on the confirmatory fork. Persona-against-generic at n=10 did not, so no claim should rest on it.

F3 is marginal. Its 40-point gap lands on the pass mark, and Fisher gives p = 0.170. Call it directional at best.

F1 is not read. Its baseline draws the blitz lane in 1 of 10 runs, under the 20% floor the gate needs. The gambler and the fortress separate by 100 points on it, and the protocol still says a fork with a dead lane is not read for discrimination. We hold to that.

8. Deviations and engine faults

From v2 to v3, two things changed at once

The brief

Redesigned.

The referee's mode

Own-difficulty mode: it sets each check's difficulty itself.

v2 against v3

Confounded.

Inside v3

Baseline and arms share the setting. Consistent.

a comparison that cannot be read

The F1 baseline, read three times

At 5 of 10 runs

0%

Provisional liveness failure.

All 10, first tag

2 of 10

A pass at the boundary.

All 10, claim line

1 of 10

The failure, reinstated.

blitzconsolidater9, blitz in its reasoninga call we reversed

Three process lessons

The pre-registered n

Read a gate there, never at half of it.

Provisional first

Log an interim call before the data lands.

The stated rule

Score with the line the method names.

Inject delivery lag, about two ticks

v2: 4 ticks

A decision near the final turn waits on an inject that lands about two ticks late. Under v2, F3 almost never landed.

about two ticks of lag

An engine fault

The personas played no part.

8. Deviations and engine faults

D1, own-difficulty confound. The v3 launcher turns on own-difficulty mode, which makes the referee set each check's difficulty from its own reading of the action. The v2 fork runs did not. So the brief redesign and the difficulty-setting mode changed in the same increment. Any v2 against v3 comparison of lane shares is confounded. The v3 arms stay internally consistent, because the baseline and the persona arms share the setting. The lesson is to change the instrument or the engine mode, and never both at once.

D2, an interim call that was reversed, then reinstated. At 5 of 10 baseline runs the F1 blitz lane had drawn 0%, and the log recorded a provisional liveness failure. The full baseline, scored by the first tag in each transcript, gave 2 of 10, and the report of 14 August recorded a pass at the boundary. Scored from the claim line, as the method says, the full baseline gives 1 of 10. The run order was consolidate for r1 to r7, blitz for r8, then consolidate for r9 and r10. Run r9 wrote blitz in its reasoning and consolidated on its claim. So the provisional failure stood all along, and the correction of 18 September reinstates it.

Three process lessons follow. Read a gate at the pre-registered n, never at half of it. Log an interim call as provisional before the data lands, because that is what makes being wrong cheap. And score with the rule the method states, because a scorer that reads a different line from the one the brief asked for can pass a gate the data failed.

Inject delivery lag of about two ticks. This engine fault is why F3 almost never landed under v2, and the personas played no part. It affects any brief that puts a decision near its final turn.

9. What this licenses

Supported

Opposite branches

At the same genuine fork, in the direction written down first. Read mechanically.

Not supported

A real stakeholder's branch

Any persona picking it.

Outside this brief

The effect size.

Persona against generic

Underpowered here.

The two secondary forks

F1 not read. F3 on its mark.

what the test showednot shown

9. What this licenses

Supported: different personas take opposite branches at the same genuine fork, in the direction written down beforehand, with the branch read mechanically from the transcript.

Not supported: that any persona picks the branch a real stakeholder would pick. That the effect size holds outside this brief. That persona-against-generic separation is established, because that comparison is underpowered here. That the two secondary forks separate their pairs, because F1 is not read and F3 sits on its mark at p = 0.170.

Back to the validation summary, the plain-language account of the four published studies and the fifth we withdrew.

The annex · Deep

Where we wrote down the rules for this study before it ran, and where the run data sits.

FieldValue
Pre-registrationthe study protocols, dated 10 August 2026
Datathe run data, in our private repository
Accessthis repository is private. Pre-registration documents and run data are available on request.

Back to the evidence page for what the four published studies found together, the fifth we withdrew, and the limits that apply to every one of them.