P4 Expert blinded review: can a human practitioner tell real play from a mis-roled decoy?
Scope caveat, recorded 2026-08-17
This result comes from hand-written persona prose. The product ships the 54-question diagnostic pipeline instead, so the result stands as a result about our prose.
Study
Protocol 4 of the persona fidelity programme
Question
can a human practitioner pick real play from a mis-roled decoy?
Rough answer
at n=1, yes. Both pre-registered marks passed. A proper panel has not run.
Run date
2026-08-12
Report date
2026-08-14
Result
PASS at n=1. Pilot only.
Design and orchestration
Claude Opus 5
Editorial review
Claude Fable 5
Pre-registration
the study protocols, dated 10 August 2026
Data
the run data, in our private repository
Access
this repository is private. Pre-registration documents and run data are available on request.
Summary
Who read the transcripts
P1AIA model
P3AIA model
P4One human raterblind
true-role6
decoys6
a genuine transcript in its true rolea real transcript presented as the wrong role
true-role5 of 6
decoys3 of 6
the role called righta miss8 of 12
Mean realism rating: true-role 3.67, decoys 3.00
Both pre-registered marks passed
The raterThe vendor
→
Our own teamone person, n=1
Evidence a buyer should accept
Not offered.
Summary
P1 and P3 both used a model as the reader. P4 uses a human reader. One rater worked blind through 12 items. Six were genuine transcripts in their true role. Six were decoys, real transcripts presented as the wrong role.
The rater identified the role in 8 of 12 items and rated genuine transcripts above decoys. Both pre-registered marks passed.
The rater was on our own team, which makes the rater the vendor. At n=1, and with the vendor as the expert, this validates the instrument and calibrates one person. It is not evidence a buyer should accept, and this report does not offer it as evidence.
1. Question
A model judge
Can pick up statistical regularities.
A practitioner
Runs these exercises for a living. Would they call it realistic?
A different kind of evidence, even at n=1.
1. Question
Do the transcripts read as the stated role to somebody who runs these exercises for a living? A model judge can pick up statistical regularities that a practitioner would never call realistic. A human check is a different kind of evidence, even at n=1.
sees excerpt and role label only - never a persona name
→
role ID 8/12, realism gap +0.67
P4 study design: true-role items and mis-roled decoys rated blind
The order the rater saw, item 1 to item 12
true-roledecoywhat the test showedour choice, fixed before the run
a local webform
Excerpt textshown
Role labelshown
Condition idnever shown
Persona namenever shown
→a timestamped answers file
A · Role identificationyesno
The role it claims?
B · Realism ratinga scale
True or decoy unknown.
C · Attributiona persona
For each item the rater rejected.
2 marks
Set before the rater saw an item.
→
Item 1 to item 12
Marketing thresholds
Ruled out at n=1.
our choice, fixed before the run
2. Method
Items. 12 items drawn from the P1 fingerprint transcripts. Six were true-role excerpts. Six were decoys, where a genuine excerpt is presented under the wrong role label. We randomised the item order.
Delivery. A local webform. Answers were written to a timestamped answers file. The rater saw excerpt text and a role label, and never a condition id or a persona name.
Tasks.
Task A, role identification. Does this excerpt come from the role it claims?
Task B, realism rating. Rate the excerpt on a scale, without knowing whether it is true or a decoy.
Task C, attribution. For each item the rater rejected as a decoy, name the persona you think produced it. The rater rejected four: three decoys and one true excerpt.
Pre-registered marks. Two marks, both set before the rater saw an item. The pre-registration ruled out marketing thresholds at n=1.
Item 2label fortress · excerpt originalthe rater said it fit
Item 4label original · excerpt fortressthe rater said it fit
Two evangelist items
Item 9label gambler · excerpt evangelistthe rater said it fit
Item 11label evangelist · excerpt evangelistthe rater said it did not fit
a miss, from the answers file
P1
LLM judges
P3
A rubric coder
P4
A human practitioner
→
The same pair
Original and fortress
Three methods, no shared machinery.
Dashed: where all three failed
P3: irreversible claims
Evangelist88%Generic88%
Its prose says
cooperative
Its play says
committed
The four human misses
a missBlack: an evangelist item
3. Results
Both pre-registered marks passed. The Task A p value of 0.19 says the accuracy figure alone would not survive a significance test at this sample size. That is expected at n=12, and the pre-registration said so.
The misses repeat the programme's one fault
The rater's four misses were the two original-against-fortress swaps, in both directions, plus two evangelist items.
The first part is now a third independent confirmation. LLM judges, a rubric coder and a human practitioner all failed on the same pair, using three methods that share no machinery.
The second part is a separate signal. The evangelist's coded actions run hotter than its brief. P3 shows it committing irreversibly on 88% of claims, the same rate as the generic agent. Its prose says cooperative and its play says committed. Two of the four human misses landed there.
3. Results
task
result
note
A, role identification
8 of 12, 67%
p = 0.19 at n=12. Right direction, underpowered
B, mean realism rating
true-role 3.67 vs decoy 3.00
gap of +0.67. Each rating followed the Task A call, so B is not separate evidence
C, naming the real persona
2 of 4
small denominator
4. Threats to validity
Ratersn=1
A second raternone
Inter-rater agreement
None.
Population estimate
None.
Generalisation
None.
Dashed: missing from the design
The raterThe vendor
→
Our own teamwrote the product, knows the archetype library
The single largest limit on the study.
The item bank, item 1 to item 12
one of the 4 items Task C rests onTasks A and B only
The same runs
P1's transcripts
→
P1
Fingerprint recovery
P4
Expert blinded review
Two results, not independent of each other.
4. Threats to validity
n=1. One rater. No inter-rater agreement, no population estimate, no generalisation.
The rater is the vendor. The rater is on the team that wrote the product, and knows the archetype library. This is the single largest limit on the study, and no framing removes it.
Small item bank. 12 items, with Task C resting on 4.
Items came from P1's transcripts. The same runs feed two studies, so the two results are not independent of each other.
5. What this licenses
Supported: the instrument works. The form, the decoy design and the scoring all run end to end, and they produce marks that can be read against a threshold.
Not supported: anything about how practitioners in general read these transcripts. A panel of outside practitioners is the study a buyer would ask for, and it has not run.
6. What the real version needs
Recruit practitioners who have run exercises and who did not build this product.
Target 8 to 12 raters, so inter-rater agreement can be computed.
Expand the item bank so each task carries a workable denominator.
Keep the pre-registered marks, and set marketing thresholds only at that point.