Strategy Soup What we tested

The Long Game Project · Evidence · Study 1 of 5 · TLDR

P1 Fingerprint recovery: can a blinded reader recover the persona from play alone?

P1 Fingerprint recovery

From play alone

ACTOR-X

An anonymised transcript. Names stripped out.

→
A blinded reader

Never sees the persona text.

→
Which persona?
originalgamblerfortressevangelist

24 transcripts, four personas

Readers were right79% Chance25%

what the test showedguessing among four

PASS strong

Readers were rightpass mark 50%79%

what the test showedthe pass mark we wrote down before the run

Question

can a blinded reader tell which persona produced a transcript, from play alone?

Rough answer

yes. Readers were right 79% of the time against a 25% chance rate.

Result

PASS (strong)

Method · Validation · Study 1 of 5

P1 Fingerprint recovery: can a blinded reader recover the persona from play alone?

Scope caveat, recorded 2026-08-17

This result comes from hand-written persona prose. The product ships the 54-question diagnostic pipeline instead, so the result stands as a result about our prose. The bridge study later tested the shipped pipeline on the P2 fork. It failed with a gap of +0pp, zero percentage points between the two characters. We fixed the renderer on 2026-08-18 and the retest passed at +80pp. Detail: the validation page.

Study
Protocol 1 of the persona fidelity programme
Question
can a blinded reader tell which persona produced a transcript, from play alone?
Rough answer
yes. Readers were right 79% of the time against a 25% chance rate.
Run date
2026-08-10
Report date
2026-08-14
Result
PASS (strong)
Design and orchestration
Claude Opus 5
Editorial review
Claude Fable 5
Pre-registration
the P1 plan, dated 10 August 2026
Data
the run data, in our private repository
Access
this repository is private. Pre-registration documents and run data are available on request.
Summary

24 anonymised transcripts, six per persona, each named by a majority of three judges

named correctlynamed as another persona

19 of 24correct25%chance4.0e-8about one in 25 million

Every error sits in one pair: v1-original and v3-fortress-bureaucrat.

Summary

Four personas played the same scenario in the same role. Blinded judges then read anonymised transcripts and assigned each one to a persona. Judges were right in 19 of 24 cases, against a chance rate of 25%. The one-sided binomial p, the chance of a score that high from guessing alone, is 4.0e-8, about one in 25 million.

The persona layer shows in play. A reader who never sees the persona text can still recover it from what the agent did and said. Every error fell in one pair of personas, and that pair became the target for the rest of the programme.

1. Question
The persona

The engine writes it into each agent.

→
Play starts

The agent plays its seat.

→
Does it still show?

Not tested before P1.

If it does not, every claim further down about persona behaviour is empty.

Recoverable

The question P1 asks.

≠
Correct

A separate question. P5 went after it.

A persona a reader can recover can still be a bad model of a real stakeholder.

1. Question

The engine writes a persona into each agent. Nothing had tested whether the persona still shows once play starts. If it does not, every downstream claim about persona behaviour is empty.

The question asks only whether the persona is recoverable. Whether the persona is correct is a separate question, and P5 went after that one. A recoverable persona can still be a bad model of a real stakeholder.

2. Method

Same scenario. Only the persona in the target role varies.

4 personas

One seat.

→
24 game runs

6 each.

→
Anonymiser

And leak guard.

→
3 blinded judges

Majority vote.

→
19 of 24

Correct.

what the test showed

P1 study design: four personas, anonymised play, blinded judges

Each run: the target actor's claims and journals, every tick

Before
actor idpersona namename stringmechanical numbers
↓
After the anonymiser
ACTOR-XACTOR-XACTOR-Xmechanical numbers

kept, because every condition shares them

Leak guard, unit-tested: no persona name or actor id survives.

Three Sonnet 5 judges, one call each, per transcript

Judge 1

4 summaries, 1 transcript.

Judge 2

4 summaries, 1 transcript.

Judge 3

4 summaries, 1 transcript.

→
Majority vote

Names the persona.

Menu order

Shuffled for every call.

Never shown

The persona library. The scenario's own briefing text.

Pre-registered materials

The persona file

f27628bd...

The scenario file

a455a8e1...

→
commit d221c82

Both files as at this commit.

A reviewer with access to the repository can hash the files again and compare.

2. Method

Design. One brief, one target role, four personas in that role. Everything outside the target role's identity text stayed constant across conditions.

Materials. Each run produced the target actor's claims and journals for every tick, the engine's word for a turn. An anonymiser stripped actor ids, persona names and literal name strings, and replaced them with ACTOR-X labels. A unit-tested leak guard checks that no persona name or actor id survives. Mechanical numbers stay, because every condition shares them.

Judges. Three Sonnet 5 judges read each transcript, one call each, and voted by majority. Each judge saw four short persona summaries and one transcript. We wrote the summaries fresh for the judging task. Judges never saw the persona library or the scenario's own briefing text. We shuffled the order of the persona menu for every call.

Pre-registered materials hash. the persona file at f27628bd... and the scenario file at a455a8e1..., both as at commit d221c82. To verify, hash each file as at that commit with SHA-256 and compare. We share the repository with reviewers on request.

2. Method

itemvalue
briefenterprise software pricing, the scenario behind report 002
target roleATLAS
conditionsv1-original, v2-gambler, v3-fortress-bureaucrat, v4-evangelist
repeats6 per condition, 24 runs
agent modelHaiku 4.5
arbiter modelHaiku 4.5
failures0 of 24
3. Pre-registered analysis and pass marks

Overall: majority-vote accuracy

PASS WEAK chance 25% 50% 0255075100%

Each persona on its own

PASS WEAK 25% 33% 050100%

PASS needs both: overall at or above 50% AND every persona at or above 33%.

WEAK: above chance overall, one persona at or below 25%.

The gates leave a gap: one persona between 25% and 33% is neither PASS nor WEAK. With six transcripts per persona no score can land there, because the nearest scores are 17% and 33%.

FAIL: a binomial test cannot tell the overall score from 25%.

a pass mark we wrote down before the runchance

The planned analysis

Binomial test

Against chance.

Confusion matrix

Per persona. Where each vote went.

Judge agreement

Do the three judges agree?

3. Pre-registered analysis and pass marks

PASS: majority-vote accuracy at or above 50%, which is twice the chance rate, AND every persona individually at or above 33%.

WEAK: above chance overall, but at least one persona at or below chance.

FAIL: accuracy that a binomial test at alpha 0.05, the usual one in twenty false-alarm rate, cannot separate from 25%.

Planned analysis: a binomial test against chance, a per-persona confusion matrix showing where each vote went, and judge agreement.

4. Results

Each square is one transcript. Each dot is one judge.

majority named it correctlymajority named anotherjudge named the true personajudge named another

19 of 2479%p = 4.0e-8against 25%21 of 24unanimous

Overall

All 24 transcriptspass mark 50%79%

Each persona

v1-original33%83% v2-gambler100% v3-fortress-bureaucratexactly on the mark33% v4-evangelist100%

what the test showeda pass mark we wrote down before the run

judged correctlyjudged as another personadot size: how many transcripts

Confusion matrix: judge votes by true persona

4. Results

Overall accuracy was 19 of 24, 79%. A one-sided binomial test against 25% gives p = 4.0e-8. 21 of the 24 transcripts drew a unanimous vote from all three judges.

Against the pass marks, the overall bar is cleared with room to spare. The per-persona bar is met, and v3 meets it exactly at 33.3%. That is a flag. It is short of a failure.

Every error is the same swap. All four v3 misses went to v1-original. The one v1 miss went to v3. No error ever pointed at gambler or evangelist. The confusion sits entirely inside the pair the design placed closest together.

4. Results

conditioncorrectrate
v1-original5 of 683%
v2-gambler6 of 6100%
v3-fortress-bureaucrat2 of 633%
v4-evangelist6 of 6100%
5. Interpretation

From play alone

Judges were right79% Chance25%

×3 the chance rate

21 of 24all three judges agreed

what the test showedguessing among four

judged correctlyjudged as another personadot size: how many transcripts

Twelve ways to be wrong. Every error used two of them, the two inside one pair.

Confusion matrix: judge votes by true persona

5. Interpretation

The persona layer transmits. Judges recover the archetype from play alone, at three times the chance rate, and they mostly agree with each other.

If transmission were weak, errors would scatter across all three wrong answers. They did not scatter. They concentrated in the one pair whose written briefs differ least. The limit is how distinct the personas are.

6. Threats to validity
One model family
Sonnet 5

The judges.

Haiku 4.5

The players.

A judge from another family

The stronger design. Recorded as a caveat before the runs.

seat-price · the ATLAS role

All 24 runs

Another scenario family

No runs, so no evidence.

Four archetypes written to be far apart

These four personasupper bound79% Two personas a client would comparenot measuredlower

what the test showednot measured, expected lowerour reading

A closed menu of four
originalgamblerfortressevangelist
<
Open identification

No menu. Name it from nothing.

The judges had the easier task on the left.

6. Threats to validity

Same model family. Sonnet judged Haiku. A judge and a player from one family may share stylistic habits that inflate recovery. A judge from another family is the stronger design, and the pre-registration recorded this caveat before the runs.

One brief, one role. All 24 runs used seat-price and the ATLAS role. On this evidence alone, nothing here transfers to another scenario family.

Personas chosen for distance. We wrote the four archetypes to be far apart. That is correct for a gating test. It also means 79% is an upper bound, and the rate to expect for two personas a client would want compared is lower.

Judges saw four options. Recovery from a closed menu of four is easier than open identification.

7. Deviation from pre-registration
Draft
ATLASthe existing pair
FORGEthe existing pair
New persona 
New persona 
Executed
ATLASv1-original
ATLASv2-gambler
ATLASv3-fortress-bureaucrat
ATLASv4-evangelist

Only the target role varies. Changed and recorded before any live run.

d221c82

Pre-registration stores a bare SHA-256.

hash matches
86cd4b6

The generic condition for P3 appended.

HEAD

A naive check runs here.

mismatch, looks like tampering
Commit pin added 2026-08-12

Every pre-registration after this one pins the commit.

a check that fails

7. Deviation from pre-registration

The draft design used the brief's existing pair, ATLAS and FORGE, plus two new personas. The executed design put all four personas in the ATLAS role instead, so no two actors share a persona and only the target role varies. We made and recorded the change before any live run.

A second, process-level fault: the pre-registration stored a bare SHA-256, a file fingerprint, with no commit pin. The persona file changed later for a legitimate reason, at commit 86cd4b6, when we appended the generic condition for P3. A naive check against HEAD therefore mismatches and looks like tampering. We added the commit pin on 2026-08-12. Every pre-registration after this one pins the commit.

8. What this licenses

Supported

Recover the persona

An independent reader, anonymised play.

Not supported

Accurate models

Of real stakeholders.

Outside this brief

Or this role, or this model tier.

what the test showednot shown

8. What this licenses

Supported: an independent reader can recover the engine's personas from anonymised play.

Not supported: that the personas are accurate models of real stakeholders. That the result holds outside this brief, this role or this model tier.

9. What it changed downstream
P1

The error pattern: one confusable pair.

→ →
The persona library

Rewritten, because P1, P3 and P4 agreed.

9. What it changed downstream

P1's error pattern set the agenda for the whole programme. The confusable pair became the named target of P2's confirmatory fork. P3 measured the same pair through different machinery and found the same weakness. P4 put it in front of a human. P1, P3 and P4 all agreed, so we rewrote the persona library.

Back to the validation summary, the plain-language account of the four published studies and the fifth we withdrew.

The annex · Deep

Where we wrote down the rules for this study before it ran, and where the run data sits.

FieldValue
Pre-registrationthe P1 plan, dated 10 August 2026
Datathe run data, in our private repository
Accessthis repository is private. Pre-registration documents and run data are available on request.

Back to the evidence page for what the four published studies found together, the fifth we withdrew, and the limits that apply to every one of them.