54questions
→15dimensions, the Ingredients
The programme, August 2026: eight profiles, eight readings of each
The rules for every run written down before we spent it
The share of score variation that is real difference between organisations
Mean correlation between the trait level written into each profile and the score.
Questions shared between two dimensions, by design.
The weakest dimension. A direction and no more.
One of five interim claims we withdrew
Five dimensions measured nothing.
The same five dimensions, scored later. None of the five was dead.
Precision: about 8 points on a 100-point scale
Five passes on one organisation
Same Flavour on four of the five passes. Of the ten pairs of passes, six agree: 60 per cent. A model reading a written brief, so a best case.
The agent playing the regulator behaves like a regulator.
Five studies, August 2026. Each raised a check the next had to pass.
The fork. Sign a binding three-year exclusive, or take a shorter pilot you can walk away from.
A recognisable cautious bureaucrat.
Behaves like a real bureaucrat who ever lived.
Blockbuster in mid-2004, facing Netflix. Five moves to reproduce. Twenty runs, ten each way.
They scored the same. For three days our headline said:
The character made no difference.
Scored decisions came out the same way in almost every run, whoever was playing.
One of the most retold business stories. The model already knew it.
We do not claim it.
We do not claim that either.
A case the model cannot already know, scored against decisions it cannot have memorised.
Measures the organisation.
Turns the measurements into a character.
Plays a seat.
Every result so far used characters written by hand. This pipeline had never been inside an experiment.
How far apart two characters’ choices were on the same fork, in points out of 100
Blinded readers naming the character
Blinded readers named the character behind an anonymised transcript 79 times in 100. Guessing would get 25. Study 1.
Can you tell the characters apart?
Yes, clearly.
Do they change what agents decide?
Yes.
Does a personality make an agent behave like the real organisation?
Unknown. Our first test could not read it either way.
Does the diagnostic we ship drive that behaviour?
Yes, now. Our first test said no, and fixing why made the product better.
The instrument
The diagnostic asks 54 questions and scores an organisation on 15 strategic dimensions, the Ingredients as the rest of this site calls them. In August 2026 we ran a measurement programme against it: eight purpose-built organisation profiles, eight independent readings of each, and the rules for every run written down before we spent it.
What worked.
The instrument separates organisations reliably. Its mean ICC is 0.84 across the 15 dimensions. ICC, the intraclass correlation, is the share of score variation that is real difference between organisations rather than noise, so 0.84 means most of what the diagnostic reports is real. All fifteen dimensions read their own trait, with a mean correlation of 0.88 between the trait level written into each profile and the score the instrument gave it, which shows a model can read a brief back, and no more than that. No two dimensions share a question, by design. The weakest dimension, breadth of scope, gives a direction and no more. Read it that way.
What we retracted.
Five interim claims we made during the programme were wrong, and we withdrew them. One was an early claim that five dimensions measured nothing. We had judged each dimension on test cases that held it constant, so the zero result came from our test design, not from the dimensions. Four of those five dimensions later scored ICC 0.75 to 0.96. The fifth did not reach that band, and none of the five was dead.
The ceiling.
These numbers measure a model reading a written brief, which is a very consistent respondent. A human answering the same questions would be less consistent, so every figure here is a best case and no estimate of human test-retest reliability. Scores are not precise beyond about 8 points on a 100-point scale. Ask the instrument about the same organisation five times and it lands on the same Flavour, the type the diagnostic sorts an organisation into, on four of the five passes. Compare every pair of passes and the agreement is 60 per cent. That consistency caps how right any reading built on the instrument can be, and we would rather measure that ceiling than assume it.
The players, in short
Our wargames seat AI agents as the other parties: the regulator, the competitor, the board. The whole exercise rests on one claim, that the agent playing the regulator behaves like a regulator. Until August 2026 nobody had tested that, including us. We ran five studies. Each one raised a check the next had to pass, and that kept us testing for another week.
- Can you tell the characters apart? Yes, clearly.
- Do their personalities change what they decide, as well as how they talk? Yes.
- Does giving an agent a personality make it behave more like the real person? Still unknown. Our first test could not tell either way.
- Does the diagnostic we ship actually drive that behaviour? Yes, now. Our first test said no, and fixing why made the product better.
The first two are worth something. The third is the one people assume, and it is still open. The fourth is the fault we are most glad we found.
What worked
You can tell the characters apart.
We took transcripts of the AI playing, stripped every name out and gave them to blinded readers: three copies of Claude Sonnet 5, a bigger model from the same maker as the Claude Haiku 4.5 that played, and none of them was shown the character material. Those readers named the right character in 19 of 24 transcripts, 79%, against a 25% guessing rate for four characters. A result that strong would come up by chance about once in 25 million tries. In 21 of the 24 transcripts all three readers agreed. A human expert also passed a related blind test, telling real play from decoys (study P4), but that expert is on our own team and was a single rater, so that pass checks the test design and nothing more.
The characters change what they decide, as well as how they talk.
We gave the agents a genuine business fork with two defensible answers: sign a binding three-year exclusive, or take a shorter pilot you can walk away from, which costs a bit more later. We wrote down beforehand which character should pick which. The commercially-minded character, the merchant in the study reports, signed the binding deal 9 times out of 9 (its tenth run reached the fork but gave no choice marker, so it was not counted). The cautious, process-bound character, the fortress, refused it 10 times out of 10 and took the pilot. No overlap. A split that clean would come up by chance about once in 90,000 tries.
That fork exists because of an earlier fault.
Every one of our first checks made the same mistake: none could tell our standard incumbent character from our cautious bureaucrat. When all three blinded readers make the same error, the fault sits in the material. The cause was simple. The two characters differed only by degree, one cautious and the other more cautious. We rewrote both so they differ in kind, then built a test to tell them apart. The pair nobody could separate is now the cleanest separation we have.
The accuracy test, and the headline we withdrew
Distinct and accurate are different things. An agent can be a recognisable cautious bureaucrat and still behave like no real bureaucrat who ever lived.
So we ran the test. We rebuilt a documented situation, Blockbuster in mid-2004 facing Netflix, using only what was public at the time. We scored a researched character against a plain competent agent on reproducing the five moves Blockbuster made. Twenty runs, ten each way. They scored the same. For three days we reported that as a clean negative: the character made no difference.
Then we checked the test itself and withdrew that reading.
Four of the five scored decisions came out the same way in almost every run, whoever was playing. Blockbuster against Netflix is also one of the most retold business stories there is, so the model already knew the ending. A test like that can only say “we could not see anything”. It cannot say “there is nothing to see”.
The accuracy question is still open, in both directions. We do not claim the characters make agents accurate, and we do not claim they fail to. We are building a sharper test: a case the model cannot already know, scored against decisions it cannot have memorised. We wrote down in advance that a null was possible and what it would mean. We did not anticipate a test too blunt to measure anything either way. Both facts are on the record.
The fault we found in our own product
Partway through, one check caught every study above: none of them had tested the product we ship. Every result so far used characters we wrote by hand. The product builds its characters from the 54-question diagnostic, and that pipeline had never been inside an experiment.
So we ran the missing test. Same business fork, same rules, with characters built by the diagnostic instead of by hand. They failed completely. The two diagnostic-built characters made the same choice as each other on the fork, a gap of zero, while the hand-written pair kept its full split in the same session.
The diagnostic itself was sound. It measured the two organisations as clearly different, on the traits that decision turns on. The fault sat one step later. The code that turns those measurements into the character’s playing instructions was throwing the measurements away and using a stock template instead.
We fixed that step, changed nothing else and re-ran the same test. In the runs we made, at most ten per character, the two diagnostic-built characters now sit 80 points apart on the same decision, 90% against 10%. That is a narrower split than the hand-written pair’s 9 of 9 against 0 of 10, and we say so. If we had shipped without that test, every claim above would have been true of our demo material and false of the product. That is why the failures are on this page.