Study

Protocol 1 of the persona fidelity programme

Question

can a blinded reader tell which persona produced a transcript, from play alone?

Rough answer

yes. Readers were right 79% of the time against a 25% chance rate.

Run date

2026-08-10 ·

Report date

2026-08-14

Result

PASS (strong)

Design and orchestration

Claude Opus 5 ·

Editorial review

Claude Fable 5

Pre-registration

docs/plans/2026-08-10-fingerprint-recovery-plan.md

Data

docs/data/fingerprint-2026-08/

Access

this repository is private. Pre-registration documents and run data are available on request.

Summary

Four personas played the same scenario in the same role. Blinded judges then read anonymised transcripts and assigned each one to a persona. Judges were right in 19 of 24 cases, against a chance rate of 25%. The one-sided binomial p, the chance of a score that high from guessing alone, is 4.0e-8, about one in 25 million.

The persona layer shows in play. A reader who never sees the persona text can still recover it from what the agent did and said. Every error fell in one pair of personas, and that pair became the target for the rest of the programme.

1. Question

The engine writes a persona into each agent. Nothing had tested whether the persona still shows once play starts. If it does not, every downstream claim about persona behaviour is empty.

The question is deliberately narrow. It asks whether the persona is recoverable. Whether the persona is correct is a separate question, and P5 went after that one. A recoverable persona can still be a bad model of a real stakeholder.

2. Method

Design. One brief, one target role, four personas in that role. Everything outside the target role's identity text stayed constant across conditions.

Materials. Each run produced the target actor's claims and journals for every tick, the engine's word for a turn. An anonymiser stripped actor ids, persona names and literal name strings, and replaced them with ACTOR-X labels. A unit-tested leak guard checks that no persona name or actor id survives. Mechanical numbers stay, because every condition shares them.

Judges. Three Sonnet 5 judges read each transcript, one call each, and voted by majority. Each judge saw four short persona summaries and one transcript. We wrote the summaries fresh for the judging task. Judges never saw the persona library or the scenario's own briefing text. We shuffled the order of the persona menu for every call.

Pre-registered materials hash. fingerprint-personas.ts at f27628bd..., seat-price.ts at a455a8e1..., both as at commit d221c82. To verify from a clone of the repository, which we share on request:

git show d221c82:app/experiment-briefs/fingerprint-personas.ts | sha256sum

3. Pre-registered analysis and pass marks

• PASS: majority-vote accuracy at or above 50%, which is twice the chance rate, AND every persona individually at or above 33%.
• WEAK: above chance overall, but at least one persona at or below chance.
• FAIL: accuracy that a binomial test at alpha 0.05, the usual one in twenty false-alarm rate, cannot separate from 25%.

Planned analysis: a binomial test against chance, a per-persona confusion matrix showing where each vote went, and judge agreement.

4. Results

Overall accuracy was 19 of 24, 79%. A one-sided binomial test against 25% gives p = 4.0e-8. 21 of the 24 transcripts drew a unanimous vote from all three judges.

Against the pass marks, the overall bar is cleared with room to spare. The per-persona bar is met, and v3 meets it exactly at 33.3%. That is a flag. It is short of a failure.

Every error is the same swap. All four v3 misses went to v1-original. The one v1 miss went to v3. No error ever pointed at gambler or evangelist. The confusion sits entirely inside the pair the design placed closest together.

5. Interpretation

The persona layer transmits. Judges recover the archetype from play alone, at three times the chance rate, and they mostly agree with each other.

The failure mode is informative. If transmission were weak, errors would scatter across all three wrong answers. They did not scatter. They concentrated in the one pair whose written briefs differ least. The limit is how distinct the personas are. Transmission works.

6. Threats to validity

• Same model family. Sonnet judged Haiku. A judge and a player from one family may share stylistic habits that inflate recovery. A judge from another family is the stronger design, and the pre-registration recorded this caveat before the runs.
• One brief, one role. All 24 runs used seat-price and the ATLAS seat. On this evidence alone, nothing here transfers to another scenario family.
• Personas chosen for distance. We wrote the four archetypes to be far apart. That is correct for a gating test. It also means 79% is an upper bound, and the rate to expect for two personas a client would want compared is lower.
• Judges saw four options. Recovery from a closed menu of four is easier than open identification.

7. Deviation from pre-registration

The draft design used the brief's existing pair, ATLAS and FORGE, plus two new personas. The executed design put all four personas in the ATLAS role instead, so no two actors share a persona and only the target role varies. We made and recorded the change before any live run.

A second, process-level fault: the pre-registration stored a bare SHA-256, a file fingerprint, with no commit pin. fingerprint-personas.ts changed later for a legitimate reason, at commit 86cd4b6, when we appended the generic condition for P3. A naive check against HEAD therefore mismatches and looks like tampering. We added the commit pin on 2026-08-12. Every pre-registration after this one pins the commit.

8. What this licenses

Supported: an independent reader can recover the engine's personas from anonymised play.

Not supported: that the personas are accurate models of real stakeholders. That the result holds outside this brief, this role or this model tier.

9. What it changed downstream

P1's error pattern set the agenda for the whole programme. The confusable pair became the named target of P2's confirmatory fork. P3 measured the same pair through different machinery and found the same weakness. P4 put it in front of a human. P1, P3 and P4 all agreed, so we rewrote the persona library.

Back to the validation summary, the plain-language account of the four published studies and the fifth we withdrew.