NVIDIA-affiliated RTL repair audit: skipping one clock cycle flips a spec's effect from -11.4 to +7.6 points, and Opus 4.6 scored 70% with no broken code at all
On 79 hardware-repair cases, how you score reset and the first cycle changes the sign of the result. Verdict: if you cite an LLM Verilog-repair score, ask which oracle and whether a spec-only baseline was run.
simulation and formal scoring give opposite signs for the apparent effect of providing a specification in one evaluated cohort; reset-aware rescoring attributes almost all of the simulation-formal disagreement to reset and first-cycle comparison semantics.
Comprehensive take
A benchmark delta smaller than the gap between two reasonable oracles is noise. Before a repair score changes a purchasing or tooling decision, demand the reset semantics and a spec-only baseline, and re-score with a formal oracle where you can.
Use cases
Chip-design teams evaluating LLM assistants
Ask vendors for reset, first-cycle and unknown-value semantics, plus a spec-only baseline, before trusting a repair percentage.
Benchmark authors
Adopt the four reporting controls and publish oracle coverage and undecided cases alongside the headline score.
Researchers
Use the released hash-bound dataset and harness to repeat the audit with repeated samples and more models, which the paper lists as future work.
Specs
| RTLRepairDataset | 5,784 training pairs source |
|---|---|
| RTLRepairBench | 459 held-out cases (374 syntactic, 85 behavioral) source |
| Spec effect, GPT-5.5, simulation cycles 0-199 | -11.4 points (75.9% vs 64.6%) source |
| Spec effect, cycles 1-199 and formal | +7.6 and +8.9 points source |
| Spec-only minus spec+code range | -8.5 to +18.3 points across six models source |
Insights
- Axion reading: the paper shows evaluation fragility, not that any named model is better or worse at hardware repair.
- Axion note: with one greedy draw per arm and clustered cases, small gaps in these tables should not be over-read.
The setup
Yeyin (Eva) Zhu, who lists an NVIDIA affiliation, and independent researcher Rishabh Ranawat, built RTLRepairDataset (5,784 broken/fixed hardware-design pairs) and RTLRepairBench (459 held-out cases: 374 syntactic and 85 behavioral mutations). The paper, a workshop submission in the AI for Chip Design track, uses them to ask a less glamorous question than who scores highest: what does a repair score actually measure?
Finding one: the scoring contract can flip the answer
On 85 behavioral cases the team tested GPT-5.5 under two prompts, one giving a tool diagnostic and one adding the specification. They scored the same outputs four ways, restricted to the 79 cases with definitive formal-equivalence verdicts. Simulation over cycles 0 to 199 says adding the specification hurts: 60 of 79 passes (75.9%) with the diagnostic versus 51 of 79 (64.6%), a contrast of -11.4 points. Skip only the first compared cycle and the picture reverses: 72 of 79 (91.1%) versus 78 of 79 (98.7%), +7.6 points. Reset-aware simulation and formal equivalence both give 89.9% versus 98.7%, +8.9 points.
Reset-aware rescoring cleared all 39 cases where simulation said fail but formal proof said equivalent. Across all 85 cases it cleared 12 of 12 and 27 of 30 in the two arms. The authors say reset and first-cycle semantics explain almost all, not all, of the disagreement. If a benchmark scores by simulation match, those semantics are part of the measurement definition.
Finding two: success is not proof of repair
On 274 mined failures, each model got broken code without the spec, broken code with the spec, or the spec alone. Spec-only minus spec-plus-code ranged from -8.5 to +18.3 points across six models. GPT-5.5 scored 81.5% with code and spec versus 76.9% spec-only. Claude Opus 4.8 scored 71.6% versus 69.0%. Claude Opus 4.6 is the outlier: 9.5% without the spec, 51.6% with both, and 70.0% from the spec alone, +18.3 points over repair.
The authors are careful: this does not show a model ignored the faulty code, only that a correct output cannot tell you whether it repaired or rewrote. Spec-only recovery exceeded the no-spec arm for every model.
Caveats you should weigh
This is an exploratory study. The 79-case comparison uses one greedy draw per arm, the cases share source designs, and the authors decline to report p-values because cases cluster within 112 source tasks. The paper also states that model labels are historical endpoint snapshots, not reproducible provider revisions. Treat the sign flip as a demonstration of fragility, not a population estimate.
What to do with it
Four controls follow: separate syntactic from behavioral bugs, state initialization, reset, first-cycle and unknown-value semantics, include a spec-only baseline, and report oracle coverage and undecided cases. The paper's closing warning applies to buyers of any LLM EDA tool: generated RTL still needs independent simulation, formal analysis where supported, and human review. Code and data are at github.com/Evaaaz/RTLRepairBench.