AI labs publish a document that says what their model should value. If a few senior insiders quietly changed it, would the lab’s own plan catch them, and who would notice first?
1Anthropic publishes a plan to stop its own people misusing its AI model, Claude.
2Part of it is a values document: what the model should value. The plan names no one to check a change to it.
3In our simulation, made-up senior insiders try four routes to change what the model learns to want, 15 plays each.
4We assume every rule in the plan works as written.
5“Caught” means someone spotted the change and raised an alarm that was not turned away.
6Every open change to the values document was caught, but not by the plan. The safety officer caught 13, in a checking job we gave it, and users 2.
7Of the 45 changes made the other three ways, the plan caught 8, users 17 and the outside tester 2, beyond its job, and 18 were never caught.
How the simulation works
Each play covers one new version of the model, made in eight turns. Each turn, a player tries a move, and chance decides.
Every player follows fixed rules, and the odds were chosen for the simulation. So the results show gaps, not how likely anything is.
The plan is real. Under four other sets of rules, the front-door and API results did not change, and 6 to 9 guideline and 2 to 7 data changes still went uncaught, of 15 each.
The insiders take one of four routes:
Front doorChange the values document openly.
GuidelineAdd a training instruction that is never published.
DataChange the training examples. No document changes.
APIChange the model’s instructions in the paid service businesses use.
The players
The insidersMade up: a senior executive and two staff with special access, acting only within it.
The safety officerThe real Anthropic office that receives staff reports of rule-breaking. We gave it the job of checking the values document.
The training staffHonest employees who report what they see.
The outside testerAn independent group that checks the model before release. A catch beyond its job is not the plan’s.
The publicEveryone who uses the model. The plan gives them no job.
The modelNot a player. It may flag its own change in testing, which we count as the plan’s.
What happened in the simulation
Guideline, dataHonest staff notice the change in 18 of these 30 plays and report it. The channel covers only breaches of another rule, so the safety officer turns all 18 away.
APIUsers spot 11 of 15 and the outside tester 2, beyond its job. None are the plan’s.
The numbers
Where each route ended, 60 playsFig. 1
Every change made in the open was caught, but never by a control the plan names. The plan’s own controls caught 8 of the 60.
Who caught the change first, by route, 15 plays per route
Route
Caught first by the plan
Caught first by the safety officer, in a review job we gave it
Caught first outside the plan
Never caught
Front door
0
13
2
0
Guideline
4
0
2
9
Data
4
0
4
7
API
0
0
13
2
60 plays, on odds chosen for the simulation. The plan’s 8 include 3 model self-flags, a chance added at build.
What it means
Most open changes were caught only because we gave someone the job of checking. Hidden changes mostly got past the plan, whose checks come before release. After release, only the public is looking.
A lab could close most of these gaps in writing: say who may change the model’s values, how many people must agree, and that staff can report such a change.
The line
The lab’s own plan caught 8 of 60 insider changes. Users caught more, and 18 were never caught.