Plain coding-agent session beat four specialised ML-engineering harnesses: 62.5% vs 47.1% medal rate on GLM 5.2, at 6.2x the token cost (EPFL/Apple preprint)
Matched backbone, hardware and time, a minimal read/write/bash agent matched or beat every harness on all 17 MLE-bench pairs tested. Verdict: spend on the model and runtime first, but budget for the context bill.
We find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance.
Comprehensive take
If you are building or buying an MLE agent, start with a strong model in a real coding-agent runtime (shell plus filesystem) and measure before adding orchestration. The paper gives no evidence that tree search, populations or multi-agent layers pay for themselves at frontier backbones. It also shows the minimal design is token-hungry, so price the context before you commit, and treat the result as strong preprint evidence rather than settled fact.
Use cases
Teams evaluating an agent harness for Kaggle-style ML work
Run a plain single-session coding agent on your own tasks as the control. On the paper's tasks the four published harnesses never significantly beat it with frontier backbones.
Anyone budgeting inference for long agent runs
At GLM 5.2 the single long session cost a modelled $12.12 per run against $1.95 for AiScientist, driven by cache-read tokens. Hardware ($24 to $85 for 24h on an A100 80GB) still dominates the total.
Builders of agent frameworks
The strongest gain was moving from chat to a coding-agent environment. Invest in the runtime and tool primitives before search structure.
Specs
| Medal rate, GLM 5.2 (MLE-bench) | Malena 62.5% vs best external harness 47.1% source |
|---|---|
| Scope | 17 harness-backbone pairs on 30 MLE-bench tasks; 6 pairs on 40 NatureBench tasks source |
| NatureBench surpassed-SOTA, GLM-5.2 | Malena 21.7% [15.0, 27.5]; AiScientist 17.5%; MLEvolve 10.8% source |
| NatureBench surpassed-SOTA, Kimi-K3 | Malena 26.7%; AiScientist 27.1%; MLEvolve 11.7% source |
| Modelled cost per run, GLM 5.2 | Malena $12.12 vs AiScientist $1.95 (about 6.2x) source |
| Hardware cost, 24h A100 80GB | $24 to $85 depending on provider source |
Insights
- Axion reading: the practical takeaway is a baseline discipline. Any new harness claim should be compared against a one-session coding agent on the same model and time budget.
- Axion caution: the authors compared their own baseline against harnesses they re-ran, so tuning asymmetry is possible; the paper documents fixes in its appendix but this is not independent replication.
- Axion note: on Kimi-K3 NatureBench the Malena and AiScientist figures (26.7% vs 27.1%) are a tie, so the result is no-worse-than rather than always better.
What the paper tests
Autonomous machine-learning-engineering (MLE) agents have been getting more elaborate: search trees over candidate solutions, populations of competing programs, retrieval subagents, teams of specialised workers. The authors, from EPFL and Apple (the paper notes the work was done during an internship at Apple), ask which of that machinery still earns its keep once the underlying model can run code, read files and debug inside a single long session.
They hold the backbone, hardware and time budget fixed and climb a ladder of harness interventions re-implemented in one codebase: a chat loop, a coding-agent environment with read, write and bash primitives, search strategies, more autonomy, and multi-agent orchestration. The end of the ladder is Malena, a deliberately minimal baseline: one well-prompted coding-agent session with a small tool set.
The headline result
Giving the model a shell and a filesystem instead of a chat window is, in the authors' words, by far the largest effect they measure. Beyond that, no further intervention significantly improved performance. They then compared Malena against four open-source harnesses built for this task (MLEvolve, AiScientist, Arbor and ScienceFlow) at matched compute, time and backbone. Across 17 harness-backbone pairs on 30 MLE-bench tasks and 6 pairs on 40 NatureBench tasks, Malena matched or beat every harness on every tested pair. The one exception in direction is the small Gemma 4 31B backbone, where MLEvolve is ahead by mean estimate, though the medal-rate confidence intervals still overlap.
With GLM 5.2 as the backbone, Malena earned a medal on 62.5% of competitions against 47.1% for the best external harness. On NatureBench (share of tasks where the best run surpasses the published state of the art), GLM-5.2 gave Malena 21.7% against 17.5% for AiScientist and 10.8% for MLEvolve; with Kimi-K3 the figures were 26.7%, 27.1% and 11.7%, a statistical tie between Malena and AiScientist.
The cost the abstract does not lead with
Minimal is not cheap. Because Malena is one ever-growing session that resends its prior context every turn, its modelled inference cost at GLM 5.2 was $12.12 per run, about 6.2 times AiScientist's $1.95, even though input and output token volumes are comparable. The difference is almost entirely cache-read tokens. The authors note that cost is still dominated by hardware: 24 hours of A100 80GB compute totals between $24 and $85 depending on provider. So the harness-free design costs more in tokens but, on their numbers, still a minority of the bill.
Why it might hold
Their trace analysis finds that the agent carries out the search the harnesses hard-code: it balances rare techniques against performance-oriented refinement and reuses artifacts it has already produced, rather than needing a tree or population imposed from outside. Quality rises quickly in the first few hours and plateaus around the 5 to 10 hour mark, with later gains arriving as occasional step jumps.
What to be careful about
The authors list their own limits. Most results rely on MLE-bench, which has known task-preparation issues and is built from public Kaggle competitions, a plausible contamination target; they fixed broken tasks, ran a contamination check and added NatureBench. Some confidence intervals are wide enough that only the weaker no-worse-than claim is well supported, not a claim about effect size. Most tested interventions operate within a single worker, so sophisticated inter-agent coordination protocols are left for future work. And there is possible asymmetry in tuning effort between Malena and the external harnesses, which the paper tries to offset with backbone-specific fixes documented in its appendix. This is a preprint, a single research group, and not independently replicated.