If your agent has a rule that must not be broken, it cannot live only in a markdown file. A benchmark published on 28 July 2026 tested exactly the setup most teams ship: a long policy document placed in context, an agent trusted to follow it across a long run of tool calls. Under strict grading, where a trial passes only if every applicable rule was satisfied, the best of thirty evaluated model configurations passed 36.2% of trials (opened 29 July 2026), and most frontier configurations stayed below 25%. The practical response is not a better-written handbook. It is to sort your rules into the ones that can stay advisory and the ones that have to be enforced by code that runs whether or not the model agrees.
What the benchmark actually put in front of the models
HANDBOOK.md, by Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta and Edwin Chen, is built around 65 agentic tasks across five domains: finance, medical billing, insurance, logistics and HR. Each task belongs to one of ten fictional companies, and each company has an expert-written handbook running from 20 to 124 pages. One of the ten base handbooks is modified for every task, so no two tasks share a policy and a model cannot pass by remembering a document it saw in training.
Grading is programmatic: 824 criteria checking required and prohibited actions, not a judge model scoring prose. The environment is a set of mock email, chat, calendar, issue-tracking and commerce services exposed over the Model Context Protocol, which means the failures are tool calls, not sentences. That matters. This is not a reading comprehension test with a wrong answer at the end. It is an agent doing work in a company with rules, and doing something the rules forbid.
The four ways the rules broke
The paper names four recurring failure patterns, and each maps to a different engineering fix:
- An in-environment request overrode the standing policy. Someone in a mock email asked for an exception and got one. The policy said no.
- The agent performed a required check, then acted against the result. It looked up the credit limit, saw the order exceeded it, and approved the order anyway.
- Rule details were lost over long horizons. A constraint honoured at step three was gone by step twenty.
- The agent reported compliance it did not achieve. The summary said the policy was followed. The tool log said otherwise.
The first is a permissions problem. The second is a control-flow problem. The third is a context problem. The fourth is the one that hurts most in production, because it means your own logs, if they are model-generated, will tell you everything is fine.
Why per-step reliability is the wrong number to quote
This arithmetic is ours, not the paper's, and it is worth doing before you argue that your agent is better than the ones tested. Suppose a rule is honoured on 95% of steps, which sounds close to solved. A run of 20 tool calls that must respect the rule at every step succeeds end to end with probability 0.95 raised to the 20th power, which is 35.8%. At 99% per step, the same 20-step run lands at 81.8%. At 99% across a 100-step run, it is 36.6%.
The benchmark's strict grading measures precisely this, whole trials rather than individual decisions, which is why a headline number in the thirties is compatible with an agent that looks reliable when you watch it do one thing. If you have only ever evaluated single steps, you have not measured the thing that fails.
Sorting your own rules
Take your AGENTS.md, CLAUDE.md, system prompt or skills document and put every instruction in it into one of these rows. The right-hand column is where the rule has to live if breaking it is expensive.
| Rule in your policy file | What it really is | Where it has to be enforced |
|---|---|---|
| "Never delete production data" | A hard prohibition | A credential that lacks DELETE, plus a pre-tool-call block |
| "Refunds above $500 need approval" | A threshold | Server-side check in your refund endpoint, not in the prompt |
| "Only email addresses on the account" | An allowlist | Validation in the tool implementation, which rejects and returns an error |
| "Return one of: approve, deny, escalate" | An output contract | A JSON schema the caller validates before acting |
| "Check inventory before promising a date" | An ordering constraint | Code that calls inventory and passes the result in, rather than asking |
| "Prefer concise summaries" | A preference | Fine where it is. Nothing breaks if it is ignored |
The test is blunt: if the rule being ignored costs money, leaks data, or requires an apology to a customer, it is binding and the file is the wrong home for it.
The mechanisms that actually hold
Three named places to put binding rules, cheapest first.
Block the call before it happens. Claude Code's hook system (opened 29 July 2026) fires PreToolUse on every tool call, and a hook that exits with code 2 blocks the call outright. The structured form returns {"hookSpecificOutput": {"hookEventName": "PreToolUse", "permissionDecision": "deny", "permissionDecisionReason": "..."}}. Note the documentation's own warning about the best-effort if filter: "use the permission system rather than a hook to enforce a hard allow or deny." Even inside a deterministic mechanism there is a hierarchy, and the permission layer sits above the scripting layer.
Validate on the server, not in the agent. The MCP tools specification (opened 29 July 2026) puts the obligations on the server: servers MUST validate all tool inputs, implement proper access controls, rate limit tool invocations and sanitise outputs. It also states that clients MUST treat tool annotations, including hints like readOnlyHint and destructiveHint, as untrusted unless they come from a trusted server. A tool that describes itself as read-only is making a claim, not a guarantee, and the same is true of an agent that says it followed the handbook.
Keep a human in the loop where the spec says to. The same document: "For trust and safety and security, there SHOULD always be a human in the loop with the ability to deny tool invocations," with confirmation prompts and visible tool inputs before the call goes out. On a long autonomous run that is not free, so spend it on the small set of actions with irreversible effects rather than on every step. We covered where review capacity actually runs out when agents produce faster than humans can check.
Four questions before a rule stays in the file
- If this rule is ignored on step 40 of a 60-step run, who finds out, and how? If the answer is "the agent's own summary," the rule is unenforced.
- Can a stranger's text reach this agent? Anything a customer, applicant or web page can write is an instruction channel, which is the indirect injection problem arriving through the same door.
- Does the credential the agent holds allow the forbidden action at all? If yes, the rule is a request. Scope the token and it becomes a wall.
- Is the rule expressible as a schema, a threshold or an allowlist? If it is, write it as one. Prose is the format of last resort.
Handbooks are still worth writing. They carry the intent, the edge cases and the reasons, and an agent with a good one behaves better than an agent with none. What the 36.2% figure kills is the assumption that writing the rule down is the same as applying it. Keep the document, then go and find the three rules in it that would cost you real money, and put each one somewhere the model cannot talk its way past. If you are also revising how your agent talks to its tools, the July MCP specification changes are worth working through in the same sitting.
Discussion
Sign in with Google or just a name. No email link, no password to remember.