Do you need an agent harness? A 24-hour MLE study finds a bare coding agent matches four open-source harnesses, at roughly 6x the token cost of one
EPFL and Apple researchers ran 17 harness–backbone pairs on 30 MLE-bench tasks and found no harness that beats a plain read/write/bash agent. The catch: the simple agent costs $12.12 a run on GLM 5.2 against $1.95 for AiScientist.
Under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance.
Comprehensive take
If you run an ML-engineering agent on a frontier backbone, benchmark a plain coding-agent session against your harness before adding more machinery. Budget for context-resend cost, which is where the simple design loses.
Use cases
Baseline before you build
Run a bare read/write/bash agent on your task set first; the paper shows it is the number any harness has to beat.
Cost modelling
A single long session resends its context each turn. Price cache reads, not just output tokens, before choosing it over a resetting multi-thread design.
Small-model deployments
On Gemma 4 31B the harness MLEvolve was ahead by mean estimate, so keep workflow scaffolding if you cannot afford a frontier backbone.
Specs
| Scope | 17 harness–backbone pairs on 30 MLE-bench tasks; 6 pairs on 40 NatureBench tasks source |
|---|---|
| NatureBench surpassed-SOTA, GLM-5.2 | Malena 21.7% vs AiScientist 17.5% vs MLEvolve 10.8% source |
| NatureBench surpassed-SOTA, Kimi-K3 | Malena 26.7% vs 27.1% vs 11.7% source |
| Modelled cost per run, GLM 5.2 | Malena $12.12 vs AiScientist $1.95 (about 6.2x) source |
| Seeds at 24h | 3–4 per task (6–10 at 6h) source |
Insights
- Axion reading: the durable lesson is about attribution. Harness gains reported under different backbones and hardware cannot be compared, which is the paper's own critique of the literature.
- Axion caution: a preprint with wide intervals at 24h; treat 'no harness helps' as 'none was shown to help here'.
The question
Autonomous ML-engineering agents have grown steadily more elaborate: search trees, populations of programs, memory hierarchies, teams of specialised sub-agents. Brilliantov, Hernández-Cano and Abbé (EPFL and Apple, arXiv, 30 September 2026) ask which of those layers still earn their keep once the model can run code itself. Their baseline, Malena, is one well-prompted coding-agent session with read, write and bash tools and nothing else.
What they measured
The setup is unusually controlled. Every run gets one A100 80GB, 12 CPU cores and 144GB of RAM, with open models served locally through vLLM. The authors re-implemented each class of harness intervention in a single codebase so that only that intervention varies, then ran the released open-source systems head to head.
The biggest effect by far is giving the model an execution environment at all. Moving from a chat call that must emit a whole script to a coding agent that can explore, run and debug dramatically improved GLM 5.2 and Kimi K3, and even helped the smaller Gemma 4. After that, nothing moved the needle significantly. Four search strategies (chain, greedy, UCB1, best-of-N) overlapped; the largest gap, UCB1 over best-of-N, was 2.71 percentile points with a 95% interval of -1.44 to +6.60, which includes zero.
Against released harnesses, Malena matched or beat every system on every tested pair: 17 harness–backbone pairs on 30 MLE-bench tasks and 6 pairs on 40 NatureBench tasks. On NatureBench's surpassed-SOTA rate, Malena scored 21.7% on GLM-5.2 against 17.5% for AiScientist and 10.8% for MLEvolve. On Kimi-K3 it scored 26.7% against 27.1% and 11.7%, a statistical tie with AiScientist. The only exception was the small Gemma 4 31B backbone, where MLEvolve led on the mean, though the medal-rate intervals overlap.
The part the headline skips
Simple is not cheap. At GLM 5.2 Malena's modelled cost is $12.12 per run, about 6.2 times AiScientist's $1.95. Token volumes are comparable; the gap is cache reads, because one ever-growing session resends its whole context every turn while the other harnesses reset into short-lived threads. The authors note hardware dominates anyway: 24 hours of A100 time runs $24 to $85 depending on provider.
Their own limitations also matter. MLE-bench is built from public Kaggle competitions, so contamination is plausible; they ran code-similarity and embedding checks and found nothing anomalous, and added NatureBench for that reason. The 24-hour runs had only 3–4 seeds per task, so the confidence intervals are wide, and the authors say their claim is "no worse than", not "better by X". Multi-worker coordination is largely untested.
Axion take
This is the strongest controlled evidence yet for something practitioners suspected: as agentic post-training improves, scaffolding that compensates for weak tool use stops paying. It is a result about MLE benchmarks with frontier open-weight backbones, not about all agents. Weaker models still benefit from workflow priors, and the paper says so.