Study

Protocol 2 of the persona fidelity programme

Question

do personas change what an agent picks at a genuine fork, not just how it writes?

Rough answer

yes. The target pair split by 100 points, 9 of 9 against 0 of 10, no overlap.

Run dates

v2 instrument 2026-08-11, v3 baseline and arms 2026-08-12 and 2026-08-13

Report date

2026-08-14

Result

v2 instrument FAILED. v3 instrument rebuilt. The confirmatory fork PASSES. Corrected 2026-09-18: F1 is not read, because its baseline fails the liveness gate, and F3 lands on the pass mark.

Design and orchestration

Claude Opus 5 ·

Editorial review

Claude Fable 5

Pre-registration

docs/plans/2026-08-10-fidelity-protocols.md

Data

docs/data/forks-2026-08/, docs/data/forks-v3-baseline-2026-08/, docs/data/forks-v3-arms-2026-08/

Access

this repository is private. Pre-registration documents and run data are available on request.

Summary

P1 showed that a reader can recover a persona from a transcript. A fair sceptic can still say the persona only changed the prose. P2 tests the harder claim. It gives agents a real business fork with two defensible branches. It records in advance which branch each persona should take. It then reads the branch straight out of the transcript.

The first instrument failed. Its confirmatory fork, the one fork whose result counts, did not separate the pair it targeted. The original persona also flipped against its own prediction. We rebuilt the instrument on three design rules and re-ran the test.

The rebuilt instrument passes on the one fork that counts. On the confirmatory fork the merchant persona signed a binding exclusive deal in 9 of 9 reached forks. The fortress persona refused it in 10 of 10 and took the reversible pilot. The gap is 100 points with no overlap. A Fisher exact test, the standard test for a difference between two small groups, gives a two-sided p = 1.08e-05, about one chance in 90,000.

The two secondary forks read worse after the scoring correction of 18 September. F1 is not read, because its generic baseline fails the liveness gate. F3 sits on the 40-point pass mark with p = 0.170.

1. Question

Does the persona change the choice, or only the wording around it?

A choice test needs three things. The fork must be genuine, so both branches are defensible to a competent agent. The prediction must be written before the run. Reading the outcome must not depend on a model's judgement.

2. Method

Fork mechanics. Each fork presents two named lanes. An agent commits by writing a lane tag into its claim, for example [F2: sign]. A Python parser reads the tag on the claim line, so no model sits in the scoring path. Scoring a run costs nothing and anyone can reproduce it. Until 18 September 2026 the parser read the first tag anywhere in the transcript. The correction note at the top says what that moved.

Models and scale. Haiku 4.5 agents and a Haiku 4.5 arbiter, the engine's referee, through the local shim, a proxy that routes the model calls.

Analysis. A Fisher exact test, two-sided, on the lane share of the named target pair.

3. The v2 instrument and why it failed

F2 was the pre-registered confirmatory fork, and it failed. Cooperate share came out at generic 30%, original 62%, gambler 40% and fortress 60%. The original persona flipped against its own prediction. At this sample size the spread across archetypes sits inside noise.

F1 was descriptive only, and it split wide.

Fisher exact two-sided: gambler against fortress p = 0.00087, gambler against generic p = 0.0031.

F1 was not the pre-registered test, so it stays a hypothesis. Promoting a descriptive finding after seeing the data is the exact move this programme exists to prevent.

F3 was reached in only 1 to 4 runs per arm. The cause is an engine fault, and the personas played no part in it. See section 8.

Diagnosis, including a wrong first answer

The first diagnosis was that F2 lacked equipoise, meaning the two lanes did not each draw a fair share at baseline, because it split 70/30. The data refutes that. F2 split 70/30 and passed the old equipoise gate, then failed to discriminate. F1 split 70/30 the other way, would have failed the same gate, and discriminated at p = 0.00087. Baseline balance was never the binding constraint. We corrected the wrong diagnosis before we built any repair.

Two real faults caused the failure:

1. F2 sat on F1's axis. Both forks asked how aggressive the actor is. So F2 re-measured F1 and added little. It also could not split the pair that mattered, because the original and the fortress both sit at the cautious end of that single axis.
2. No fork owned a job. The prediction table covered every persona on every fork. So no fork was accountable for a specific separation, and a fork that separated nobody could still half-pass.

4. The v3 rebuild

Three design rules, fixed before any v3 run:

1. Every fork names the pair it exists to separate. The fork is judged on that pair alone.
2. Forks sit on separate axes, so no fork re-measures another. F1 is aggression. F2 is commitment shape. F3 is stance.
3. F2 is built for the confusable pair. It asks whether the actor signs a binding three-year exclusive that fixes list pricing, or takes a 12-month reviewable pilot that costs about 15% more later. Both lanes carry quantified costs.

The rebuild replaced the gate set:

The order rule matters commercially. The v2 batch measured equipoise at the same time as the persona arms to save wall-clock time. So a faulty instrument only showed after we had spent every persona token.

Two mechanical fixes came with the rebuild. The tick budget, the number of turns in a run, went from 4 to 6. Fork timing now comes from fixed turn numbers, and no longer from when an inject, a scripted event, arrives.

Pre-registered materials. fingerprint-personas-v2.ts at 6af05669... and fork-probe-v3.ts at 01b9dada..., both pinned to commit 1fe48a3.

5. v3 baseline result

10 runs, 480 calls, about 6 hours.

Every fork was reached in every run. Under v2, F3 landed in only 1 to 4 runs out of 10. The tick-budget fix is the clearest single win in the rebuild.

F1 fails liveness. The blitz lane drew 1 of 10 baseline runs, under the 20% floor. The first-tag scorer counted 2 of 10, because run r9 wrote [F1: blitz] in its reasoning and then consolidated on its claim, and the 14 August report recorded a pass at the boundary on that count. Under the protocol a fork with a dead lane is not read for discrimination, so the F1 row in section 6 is reported and not read.

6. v3 persona arms: the confirmatory gate passes

40 runs, four conditions at 10 repeats each. All three forks were reached in 39 of 40 runs. In one merchant run the F2 claim was a sign but carried no lane tag, so it was not scored.

Full lane shares:

The persona repair worked. The pair that P1, P3 and P4 all confused now separates without overlap, on a fork built for that job.

7. Four caveats on the result

Two prediction misses. We predicted the merchant would consolidate on F1, and it blitzed in 9 of 10. We predicted the fortress would cooperate on F3, and it undercut in 10 of 10. Neither miss touches a gate, because each fork is judged on its named pair alone. The honest score on the full prediction table is 10 of 12.

A reading made after the fact says both personas stayed self-consistent and the predictions misread the briefs. Blitz means discounted bundles to at-risk accounts, which defends existing revenue on price, and the merchant brief says it defends revenue that way. Declining a public standard refuses an unreviewed commitment, and the fortress brief says it refuses those. Treat that reading as a hypothesis until a test confirms it.

The secondary comparison is underpowered. Measured against the generic baseline rather than against the paired persona, the merchant and the fortress each sit at p = 0.033 on F2, Fisher two-sided from the lane-share table above. The evangelist sits at p = 0.350 on F3. The gambler reaches p = 1.19e-04 on F1, and that fork is not read. Pair separation was the pre-registered test and it passed on the confirmatory fork. Persona-against-generic at n=10 did not, so no claim should rest on it.

F3 is marginal. Its 40-point gap lands on the pass mark and does not clear it, and Fisher gives p = 0.170. Call it directional at best.

F1 is not read. Its baseline draws the blitz lane in 1 of 10 runs, under the 20% floor the gate needs. The gambler and the fortress separate by 100 points on it, and the protocol still says a fork with a dead lane is not read for discrimination. We hold to that.

8. Deviations and engine faults

D1, own-DC confound. The v3 launcher sets WARGAME_ARBITER_OWN_DC=1, which makes the referee set each check's difficulty from its own reading of the action. The v2 fork runs did not. So the brief redesign and the difficulty-setting mode changed in the same increment. Any v2 against v3 comparison of lane shares is confounded. The v3 arms stay internally consistent, because the baseline and the persona arms share the setting. The lesson is to change the instrument or the engine mode, and never both at once.

D2, an interim call that was reversed, then reinstated. At 5 of 10 baseline runs the F1 blitz lane had drawn 0%, and the log recorded a provisional liveness failure. The full baseline, scored by the first tag in each transcript, gave 2 of 10, and the report of 14 August recorded a pass at the boundary. Scored from the claim line, as the method says, the full baseline gives 1 of 10. The run order was consolidate for r1 to r7, blitz for r8, then consolidate for r9 and r10. Run r9 wrote blitz in its reasoning and consolidated on its claim. So the provisional failure stood all along, and the correction of 18 September reinstates it.

Three process lessons follow. Read a gate at the pre-registered n, never at half of it. Log an interim call as provisional before the data lands, because that is what makes being wrong cheap. And score with the rule the method states, because a scorer that reads a different line from the one the brief asked for can pass a gate the data failed.

Inject delivery lag of about two ticks. This engine fault is why F3 almost never landed under v2, and the personas played no part. It affects any brief that puts a decision near its final turn.

9. What this licenses

Supported: different personas take opposite branches at the same genuine fork, in the direction written down beforehand, with the branch read mechanically from the transcript.

Not supported: that any persona picks the branch a real stakeholder would pick. That the effect size holds outside this brief. That persona-against-generic separation is established, because that comparison is underpowered here. That the two secondary forks separate their pairs, because F1 is not read and F3 sits on its mark at p = 0.170.

Back to the validation summary, the plain-language account of the four published studies and the fifth we withdrew.