Playbook for Trustworthy Third‑Party Evaluations of Frontier Models

May 29, 2026 – A new playbook for trustworthy third‑party evaluations of frontier models is released, outlining how to design harnesses that reflect real‑world tool use, state tracking, and multi‑step workflows. The authors argue that the harness – the environment, tools, and budget – can change a model’s performance, so evaluation reports must explicitly state the claim being tested and the evidence that validates the result. They identify three claim buckets: capability elicitation, safeguard performance, and controlled comparison, and list validity checks such as reward hacking, refusals, contamination, broken problems, and sandbagging. The playbook cites GPT‑5.5’s improved performance when a compaction‑enabled harness preserves context, and UK AISI’s cyber‑range study where raising the token budget from 10 M to 100 M tokens boosted success rates by up to 59 %. It also highlights METR’s shared‑task framework that uses reusable scaffolds like Triframe and ReAct to produce comparable estimates across systems. The authors recommend that evaluation reports include a detailed description of the harness, the budget, and the scoring method to allow readers to assess the robustness of the claim. They caution that standardized harnesses may under‑elicit capability, so bespoke harnesses are preferable when feasible.