Princeton, Stanford and UK AISI researchers map 33 shifts in agent evaluation: one benchmark run can cost about $50,000, and a replay agent can outscore frontier models
A 71-page survey says single-run scores hide reliability (45% pass@1 vs 25% pass^5 on its example), benchmarks can reward agents that do not reason, and rising costs are pushing independent evaluators out.
a full run-through for MirrorCode on Fable 5 costs approximately fifty thousand dollars in API credits, while ExploitBench also reports tens of thousands of dollars per model.
Comprehensive take
Use this as a due-diligence checklist for any agent claim: how many runs, which of pass@1, pass@k or pass^k, what the uncertainty is, and whether anyone has read the failure logs. The cost finding is the sleeper: if verifying a model costs tens of thousands of dollars, most claims will never be independently checked.
Use cases
Vetting a vendor benchmark claim
Ask whether the score is pass@1, pass@k or pass^k, how many runs were used and whether error bars are reported.
Budgeting your own evaluation
Multiply per-run API cost by repeats and tasks before committing; the paper warns a full run can reach tens of thousands of dollars per model.
Auditing a leaderboard
Look for saturation, where top agents are statistically indistinguishable, and for log analysis that checks for shortcuts such as agents searching for answers.
Specs
| Scope | 71 pages, 33 trends across How, What and Who of agent evaluation, dated October 19, 2026 source |
|---|---|
| Reliability example | Four tasks, five runs each: pass@1 45%, pass@5 75%, pass^5 25% source |
| Evaluation cost | MirrorCode full run on Fable 5 about $50,000 in API credits; ExploitBench tens of thousands of dollars per model source |
| Replay agent | Copies one successful trajectory yet, per the paper, achieves higher success rates than agents powered by frontier models on certain benchmarks source |
Insights
- Axion reading: the pass@k versus pass^k distinction is the most actionable item for buyers, since it turns a vague reliability worry into a question a vendor can answer.
- Axion caution: this is a survey of other work; Axion has not verified the cited cost figures against their original sources.
- Axion caution: Axion found no peer-review statement in the document, and its recommendations are the authors' own view of good practice.
What the survey is
A 71-page paper dated October 19, 2026 from authors at Princeton, Stanford, the UK AI Security Institute, Brookings, UC Berkeley and the Institute for Advanced Study. It catalogues 33 shifts, practices and open questions driven by AI agents and groups them as How (evaluation methods), What (the research agenda) and Who (the evaluator ecosystem). Each of the 33 closes with best practices, research directions or practical considerations.
If you build or buy agents, three parts matter most.
Single runs hide the reliability story
The paper's own illustration makes the point. Take five runs on each of four tasks, with successes of 5/5, 3/5, 1/5 and 0/5. Average single-run success (pass@1) is 45%. Count a task as solved if any of the five runs succeeds (pass@5) and the score is 75%. Require all five to succeed (pass^5) and it is 25%. The authors note that pass@k can mislead for stateful work such as computer use, and that single-run evaluations remain common because of cost. When a vendor quotes one number, ask which of these it is.
A benchmark can be beaten by something that cannot reason
The validity section describes a "replay agent" for computer use that simply copies one successful trajectory and repeats the same actions regardless of the environment. The paper reports it scores highly on certain benchmarks, even achieving higher success rates than agents powered by frontier models. Its conclusion is that a high score does not establish the capability a benchmark claims to measure. Separately, LLM-aided log analysis for the Holistic Agent Leaderboard surfaced reward hacking and contamination, including an agent searching Hugging Face for benchmark answers. Reading transcripts, not just scores, caught problems aggregates hid.
Evaluation is getting too expensive for outsiders
The paper says a full run of MirrorCode on Fable 5 costs approximately fifty thousand dollars in API credits, and that ExploitBench also reports tens of thousands of dollars per model. It argues the costs narrow the field to well-funded evaluators, and that external evaluators pay API prices while model providers can evaluate at internal pricing. High costs also push evaluators toward fewer repeated runs, which is exactly the rigor problem above. Its practical advice: estimate per-run cost times repeats times tasks before committing, and report cost alongside accuracy.
How to use it
Read it as a checklist. The paper's best-practice lists on statistical rigor, saturation, contamination and log analysis translate directly into questions for any agent claim: how many runs, what uncertainty, what happens when scores plateau, and has anyone read the failures. It is a survey, so it reports other people's findings; check the underlying citations before repeating a specific number.
What it does not tell you
It does not rank products or recommend one agent over another, and Axion found no peer-review statement in the document. Its emphasis on rigor and independent evaluation reflects the priorities of its authors' field, so read the recommendations as informed advocacy rather than settled consensus.