Trail of Bits' regression-hunt cut false-positive findings 25% versus a bare Codex prompt, but it was tested on 21 regressions from its own patch review
The open-source patching-bugs and regression-hunt plugins ask agents to prove every claim on a live deployment. A bare agent run found the 21 known regressions at about 36% precision (21 of 58 candidates). Verdict: copy the three-arm proof now, wait for the full evaluation before trusting the numbers.
In our preliminary evaluations, regression-hunt produces 25% fewer false-positive findings than a baseline Codex agent prompted to "review the PR for regressions."
Comprehensive take
The most reusable idea is not the plugin but the discipline: schema-bound steps, live proofs, a revert arm, and written definitions of what a regression is not. Adopt that now; hold the 25% figure loosely until the larger evaluation lands.
Use cases
Maintainers triaging AI-found bugs
Run patching-bugs on a report, then regression-hunt on the resulting diff, with your threat model supplied so agents check conclusions against it.
Teams reviewing agent-written PRs
Borrow the three-arm proof: original fails, patched passes, full revert fails again, before approving a fix.
Anyone building review agents
Copy the 'inconclusive unless a live proof of concept exists' rule to stop confident but unevidenced findings.
Specs
| Plugins | patching-bugs and regression-hunt, in the trailofbits/skills repository source |
|---|---|
| Review of merged patches | 21 confirmed regressions in over 130 merged PRs (16 functional, 1 performance, 4 build/test) source |
| Baseline precision | 21 confirmed of 58 candidates in one run, about 36% source |
| Headline result | 25% fewer false positives than a baseline Codex agent; 95% CI 14% to 38% source |
| Project inputs | THREAT_MODEL.md and PATCHING_GUIDELINES.md (or AGENTS.md) source |
Insights
- Axion reading: the 21-regression test set comes from the same patch review the authors say informed the plugin design, so the 25% result may be flattering; the post calls it preliminary.
- Axion note: the post reports fewer false positives and some 'borderline' known regressions but no recall figure, so the trade between noise and missed regressions is not quantified.
- Axion note: results are the authors' own using Codex; we found no independent reproduction.
What shipped
Trail of Bits published two open-source plugins on 21 October: patching-bugs, which reproduces a reported bug, develops a fix and tests it, and regression-hunt, which checks whether a change breaks previously supported behaviour. Both come out of Patch the Planet, the firm's joint initiative with OpenAI, and draw on its review of merged patches and thousands of engineer Codex sessions. They live in the firm's public skills repository.
The data behind them
The authors reviewed every patch a Trail of Bits engineer wrote that maintainers merged upstream. An agent proposed potential regressions, each was confirmed with a proof of concept run before and after the change, and an engineer reviewed the results. That confirmed 21 regressions across more than 130 merged PRs: 16 functional, one performance and four build or test. The post says none broke a security protection. The review was noisy: the 21 real ones sat among 58 candidates in a single run, roughly 36% precision, and each extra run adds more candidates to sort.
How the workflow tries to cut noise
Four design rules stand out. Every step takes and returns a fixed schema (claim, code revision, required check, raw evidence, open questions) so the next step verifies rather than trusts a summary. Every claim needs a live proof of concept against a running instance reached the way a user would reach it, and a step that cannot produce one reports "inconclusive". regression-hunt enumerates candidates by tracing the diff through its callers, reproduces each, replays it with a fresh-context agent, then adjudicates against written rules for what is not a regression: an incomplete fix, a latent bug or a pre-existing bug. patching-bugs adds a "three-arm proof": the reproduction must fail on the original code, pass on the patched code, and fail again after a full revert, which shows the patch rather than an environment change did the work.
The result and its limits
On the 21-regression set, regression-hunt produced 25% fewer false-positive findings than a baseline Codex agent prompted to "review the PR for regressions", with a 95% confidence interval of 14% to 38%. The authors are plain about the price: strict evidence rules left some known regressions as "borderline", flagged but not confirmed. Fewer false alarms came with fewer confidently reported real regressions, and the post gives no recall figure. The evaluation is also preliminary, and the test set comes from the same patch review that informed the design. A separate patching-bugs evaluation is still being built.
Who should try it
Maintainers drowning in AI-generated reports are the target. Add a THREAT_MODEL.md (assets, trust boundaries, supported configurations, what counts as a vulnerability) and a PATCHING_GUIDELINES.md or AGENTS.md (build, test, patch size and regression-test requirements); the plugins feed these to each step. Teams that want a drop-in number should wait: the authors say the next versions will be shaped by trace data and better evaluations.