Qwen3.8-27B ties Claude Opus 4.8 on OSWorld V2 at 21.3% for $1.10 a task, but a hand-written seed does most of the work (Agent S4 preprint)
Agent S4 evolves its own tool interfaces and lifts a 27B open model from 5.56% to 21.30% binary success; against a like-for-like baseline, evolution adds about two tasks of 108. Verdict: copy the interfaces, wait on the evolution loop.
On OSWorld V2, Agent S4, with Qwen3.8-27B and the evolved repository of interfaces, sets a new state of the art in open-source computer-use agents, reaching 21.3% binary score, 3.8 times the same model with fixed interfaces. This is above both GPT-5.5 (13.0%) and Claude Opus 4.8 (20.6%), at about 4.3% and 1.5% of their respective costs.
Comprehensive take
Treat Agent S4 as strong evidence that interface design is a first-class lever for computer-use agents, and as weak evidence that evolving interfaces automatically beats writing a good seed by hand: in the paper's own ablation the hand-written seed is worth 3.7 points on OSWorld V2 and evolution adds 1.9 more. The 'beats GPT-5.5 and Opus 4.8 at 1.5% of the cost' line is parity on one benchmark with mixed-source costs. Should you switch? Not models; do adopt programmatic interfaces for your top five applications now, and wait for independent replication before buying the evolution loop.
Use cases
Teams running computer-use agents on a fixed app set
Replace screenshot-and-click loops with programmatic interfaces (spreadsheet ranges, PDF reading, browser fetch) and document when to use each. In the paper the hand-written seed alone moved Qwen3.8-27B from 15.74% to 19.44% binary on OSWorld V2 and cut cost per task from $1.36 to $1.11.
Anyone pricing computer-use at scale
At $1.10 per task for the evolved 27B setup versus $25.50 (GPT-5.5) and $72.40 (Opus 4.8) in the reference paper's figures, an open model with good interfaces is worth a pilot on high-volume, tolerant workflows. Expect lower partial progress than frontier models (46.90% vs 54.80% for Opus 4.8).
Scientific-software automation
On ScienceBoard the evolved repository grew from 15 to 64 functions and took success from 55.03% to 58.58%, with KAlgebra rising from 48% to 68%. Specialised interfaces matter most where clicking is a poor fit.
Specs
| OSWorld V2 binary success (108 tasks) | Evolved Qwen3.8-27B 21.30% vs Qwen CUA 5.56%, Prime Agent no repository 15.74%, seed 19.44% source |
|---|---|
| Comparators, same table | Claude Opus 4.8 batched 20.60% ($72.40/task); GPT-5.5 batched 13.00% ($25.50/task); Opus 4.8 single 18.50% source |
| Partial score | Evolved 46.90% (51.06% with scratch VM) vs Opus 4.8 batched 54.80% source |
| Cost per task, evolved 27B | $1.10 (no scratch VM); $1.27 (scratch VM); Qwen CUA $1.39 source |
| ScienceBoard success (169 tasks) | Qwen CUA 41.4%; seed 55.03% (93 tasks); evolved 58.58% (99 tasks) source |
| Repeated validation | Best first score 73.33% fell to 60.00% mean on re-run; selected repo mean 71.67% over four evaluations source |
| Code | github.com/simular-ai/agent-s (per the paper) source |
Insights
- Axion reading: the like-for-like gain from evolution on OSWorld V2 (19.44% to 21.30%) is two tasks of 108 in a single reported run; the 3.8x headline compares against a different harness.
- Axion arithmetic: 21.30% is 23 of 108 tasks and Opus 4.8 batched at 20.60% is within one task-equivalent (0.93 points), so we read it as parity, not a win.
- Axion caution: the GPT-5.5 and Claude cost figures come from the OSWorld V2 reference paper, not from the authors' own runs, and we found no independent replication yet.
What Agent S4 actually does
Computer-use agents are usually limited by the interfaces a human hands them: click, type, screenshot, maybe a shell. Agent S4 (code at github.com/simular-ai/agent-s, continuing the Agent S line) asks whether the interface layer can be built by an agent instead. A meta-agent reads trajectories from a task agent attempting 30 synthetic OSWorld V2-style tasks, with no success labels and no verifiers, then writes and revises a repository of callable interfaces and usage notes: a LibreOffice range editor, a PDF reader that works whatever app made the file, a browser fetcher, a Lean proof checker. Population-based search keeps the repositories that score best on 15 held-out validation tasks the meta-agent never sees. The task model (Qwen3.8-27B) and its harness stay fixed throughout.
The headline, and what is underneath it
The abstract's number is real and it is in the tables: Qwen3.8-27B with an evolved repository scores 21.30% binary success on the full 108-task OSWorld V2 at $1.10 per task, against 13.00% for GPT-5.5 (batched, $25.50) and 20.60% for Claude Opus 4.8 (batched, $72.40). The paper's own conclusion repeats it: roughly 4.3% and 1.5% of those costs.
Three things change how much weight that sentence can bear.
First, the "3.8 times" is measured against the Qwen CUA baseline, a different harness, at 5.56% (6 of 108 tasks). The paper's own ablation (Table 4) runs the same model in the Prime Agent harness with no repository and gets 15.74% (17 tasks). The hand-written seed repository lifts that to 19.44% (21 tasks). Evolution then adds the last step to 21.30% (23 tasks). Against the like-for-like starting point, evolution's marginal contribution on the headline metric is about two tasks. That is the number to hold next to the 3.8x.
Second, the margin over Opus 4.8 is 0.7 points, less than one task (one task is 0.93 points). Read it as parity at roughly 1.5% of the price, not a win. The paper itself separates binary from partial score: Opus 4.8 reaches 54.80% partial against 46.90% for the best evolved repository, and the scratch-VM variant reaches 51.06%. Frontier models still make more progress per task; the evolved repository converts a similar share of it into fully completed tasks.
Third, the cost comparison mixes sources. The Qwen numbers are the authors' own runs; the GPT-5.5 and Claude costs are taken from Table 3 of the OSWorld V2 reference paper (Yuan et al., 2026). The gap is large enough that the direction will survive, but the 4.3% and 1.5% figures are not from a single controlled run.
Where the evidence is strongest
The cross-domain result is the more convincing one. On ScienceBoard (169 tasks across Celestia, ChimeraX, GRASS GIS, KAlgebra, Lean and TeXstudio) a fresh evolution run took success from 55.03% with the seed (93 tasks) to 58.58% (99 tasks), against 41.4% for Qwen CUA. The repository grew from 15 functions to 64, with a ChimeraX command interface, live GRASS GIS modules and a KAlgebra calculator that evaluates many expressions in one call. KAlgebra success rose from 48% to 68%, and mean model calls per task fell from 61.7 to 58.9. Again, most of the jump over Qwen CUA comes from the seed and harness; evolution's own increment is six tasks, but it is a plausible mechanism, specialised interfaces for applications where clicking is a poor fit.
The honest-methodology detail is the repeated-validation analysis. On ScienceBoard, two repositories posted the best first validation score (73.33%) and fell to means of 60.00% and 66.67% when re-run. The selected repository scored 73.33, 66.67, 73.33 and 73.33 across four evaluations, a mean of 71.67%. Long-horizon computer-use runs are noisy, and the authors built selection around that. Note, though, that we could not tell from the paper whether the final benchmark figures average several runs, so the run-to-run variance the authors warn about may apply to the headline too.
Scratch VM or not
A surprising finding: giving the meta-agent a scratch VM to test interfaces while building them did not help binary success. Population search with the scratch VM landed at 19.44% binary and 51.06% partial; without it, 21.30% binary and 46.90% partial. Linear search (keep refining one lineage) matched the scratch-VM binary score at 19.44% with a lower partial of 47.17%. With a 30-descendant budget per run and single runs per configuration, these differences are one or two tasks and should not be over-read.
What to do with it
If you run computer-use agents on a fixed set of applications, the transferable idea is cheap to try: give your agent programmatic interfaces (spreadsheet ranges, document readers, a proof checker) in place of screenshot-and-click loops, and write down when to use each. The paper shows that this layer, not the model, was where most of a 27B model's gap closed. The auto-evolution part is the expensive, less proven step; the seed alone delivered most of the lift in the ablation.