A bash-only Lean agent solved 7 of 12 Putnam 2025 problems versus 4 of 12 for a specialised harness on the same Gemini 3.8 Flash model, in a 12-problem, single-run test
mini-Lean-agent (arXiv 2610.17331, NeurIPS 2026 workshop) gives the model only bash. At a $2 cap it verified 7/12 at $2.09 per verified proof against AxProverBase's 4/12 at $5.24; with Astra it verified 12/12 for $15.37 in total. Verdict: a useful baseline to beat, not proof that specialised tooling is unnecessary.
We introduce mini-Lean-agent, a deliberately minimal agent for Lean tasks whose only model-visible tool is bash.
Comprehensive take
The strongest claim the data supports is narrow: on twelve Putnam 2025 problems, one run each, a bash-only agent beat a specialised harness on the same model at lower cost. That makes it a baseline worth running, not a verdict on specialised tooling in general.
Use cases
Teams building Lean or formal-math agents
Run mini-Lean-agent with your base model before adding custom tools, and compare cost per verified proof as well as solve rate.
Benchmark authors
Report a fixed-model, bash-only baseline alongside specialised systems so readers can separate model gains from harness gains.
Buyers choosing a model for proof search
The cost-versus-cap tables show Astra reaching 12/12 at $5 per problem while Fable 5.1 reached 9/12 at $10, under this harness and benchmark only.
Specs
| Benchmark | Putnam 2025, 12 problems (A1–A6, B1–B6), Lean 4.28.0-rc1 source |
|---|---|
| Gemini 3.8 Flash, $2 cap | mini-Lean-agent 7/12 ($14.64 total) vs AxProverBase 4/12 ($20.97 total) source |
| Astra, $50 cap | 12/12 verified, $15.37 total, 106 calls, 51.3 minutes source |
| Fable 5.1, $50 cap | 11/12 verified, $120.29 total, 133 calls, 99.9 minutes source |
| Venue | Accepted at the 6th Workshop on Mathematical Reasoning and AI (NeurIPS 2026) source |
Insights
- Axion reading: with 12 problems and one run per harness, the 7-versus-4 result is a three-problem difference and carries no confidence interval.
- Axion note: the paper itself flags possible training-data contamination for the models used, which weakens any claim about unseen-problem ability.
- Axion note: the Numina cost comparison mixes models (Opus 4.5 versus Astra) and budgets, so it is not a harness-only comparison.
What the paper tests
Formal theorem-proving agents for Lean often ship specialised harnesses and tools, which makes it hard to tell where a benchmark gain comes from: the base model, the harness, a particular tool, or a mix. A group led by Tomàs Ortega, with Sanjeev Arora and Alex Kontorovich among the authors, proposes a control. mini-Lean-agent exposes one tool to the model, bash. The model has to inspect and edit the repository, search Mathlib with ordinary commands such as rg or grep, run Lean with `lake env lean`, and keep its own scratch files, while a separate trusted evaluator verifies the final proof. Code is at github.com/TomasOrtega/mini-lean-agent.
The benchmark is the twelve Putnam 2025 problems (A1–A6, B1–B6) released with Numina-Lean-Agent, using Lean 4.28.0-rc1 and Mathlib at the matching version, each run in its own Docker container with web access and search grounding disabled.
Same model, different harness
Holding the model fixed at Gemini 3.8 Flash with high thinking, and capping each problem at $2, 200 model calls or 120 minutes, mini-Lean-agent verified 7 of 12 and AxProverBase verified 4 of 12. The charge per successful run was $2.09 versus $5.24, and total spend including failures was $14.64 versus $20.97. Truncating the same runs at lower caps widens the gap: at $1 per problem mini-Lean-agent still verified 4 of 12 while AxProverBase verified none, and at $0.50 neither verified any.
The paper's usage breakdown suggests why. AxProverBase made fewer calls (724 versus 1,091) but generated 3.64 million reasoning tokens against 0.70 million, and its cache reads were 8.7% of input tokens versus 91.5% for mini-Lean-agent. Memory and review calls accounted for 11.1% of AxProverBase's charge, and all six of its runs that exhausted the allowance left `sorry` in their final candidates. For the two hard problems, A5 and B6, both harnesses failed even with budgets raised to as much as $50 and 24 hours.
Stronger models
With the harness fixed and a $50 cap, Astra verified all 12 problems for $15.37 in total ($1.28 per verified proof), using 106 model calls across the suite and solving A5 for $2.69. Fable 5.1 verified 11 of 12 for $120.29 ($10.94 per verified proof including the failure), leaving only A5 unsolved after $50.10. Cost-versus-cap tables show how different the two are: Astra reached 10 of 12 at $2 and 12 of 12 at $5, while Fable 5.1 reached 1 of 12 at $2 and 9 of 12 at $10.
The authors compare Astra's 12/12 with Numina-Lean-Agent, which they say reports roughly $50 per task and, on A5 and B6, $1,000 and $300 respectively. They are careful to add that Numina used Opus 4.5 and a different budget, so this shows what whole systems achieve and does not isolate the harness. A preliminary mini-Lean-agent run with Opus 4.5 at a $5 cap solved none of the 12.
Caveats that matter
The sample is twelve problems, with one run per problem and harness, so a 7-versus-4 gap is three problems. The authors say they cannot rule out contamination because the Astra, Fable and Gemini versions used may have been trained on Putnam 2025 answers. Budgets and model versions differ across the Numina comparison, and the Opus 4.5 run used a different provider route and thinking budget. A concurrent project, h1, makes a similar minimal-harness argument but keeps a Lean library-search tool, which the paper notes mini-Lean-agent does without.
What to take from it
The practical lesson is about evaluation hygiene. Before crediting a new Lean prover's tooling for its score, rerun the same model under a bash-only agent and compare cost per verified proof, not just solve rate. The paper makes that baseline cheap to run.