Galbot's PhysRSI-eval: Claude Opus 5.5 scores higher than GPT-6.1 Sol on robot-task success (0.744 vs 0.703) but needs far more of its budget to get there
A new 18-task benchmark separates how well a model starts, how fast it improves after failure and what that costs. Public benchmark averages did not predict the physical profiles, and the authors flag six systems and one run chain per task as limits.
Systems with similar public benchmark averages can still differ widely on the five axes, and a system that reaches a high final score may improve late or consume more of its budget.
Comprehensive take
The useful idea is the five-axis split, not the ranking: if you are choosing a model to drive hardware, ask how many physical attempts it needs before it works, not only whether it eventually does. Treat every number as exploratory until someone other than the benchmark's authors reruns it.
Use cases
Choosing a model for a robot-in-the-loop agent
If physical attempts are expensive or wear hardware, the Efficiency and Budget axes matter more than the final Success score; the paper shows they can point in opposite directions.
Designing your own self-improvement evaluation
Copy the intervention design: clear the workspace to test whether retained experience helps, then freeze it on unseen layouts to test whether it transfers.
Sanity-checking vendor benchmark claims
A model that leads on coding or general public benchmarks can still fail to improve physically; ask for task-specific, budget-aware results.
Specs
| Benchmark | 18 RoboDojo dual-arm tasks, six difficulty strata, fixed physical interaction budget, frozen weights source |
|---|---|
| Opus 5.5 vs GPT-6.1 Sol, Success | 0.744 vs 0.703 source |
| Opus 5.5 vs GPT-6.1 Sol, Efficiency | 0.609 vs 0.822; budget saved on solved tasks 0.579 vs 0.808 source |
| Tasks solved of 18 | Astra 15, GPT-6.1 Sol 12, Opus 5.5 13, Sonnet 5.5 11, GPT-6 Sol 7, GPT-5.6 Sol 1 source |
| History ablation (Astra, six tasks) | Late-round success 41.7% to 8.3% without history source |
| Early-round forecast error | MAE 0.019 vs 0.112 for holding the last score source |
Insights
- Axion reading: the Opus-versus-Sol result is a reminder that final success and cost-to-success are separate purchasing criteria, which a single leaderboard number conceals.
- Axion caution: the benchmark's authors also ran it, there are six systems and one improvement chain per task, and the paper calls its bottleneck and timing claims hypotheses.
- Axion caution: the paper does not say whether differences in harness configuration, rather than the underlying model, drive the timing gaps.
What the paper actually measures
Galbot's team asks a question that most robotics-agent write-ups skip: not "how good is the model at the end", but "how fast does it get better after it fails, and what does that cost in physical attempts". Their benchmark, PhysRSI-eval, puts six model-plus-harness systems on 18 RoboDojo dual-arm manipulation tasks with frozen weights and a fixed interaction budget. Each round the agent gets one judged physical attempt, then revises code, tools and notes in a persistent workspace with robot actions switched off. A seventh system, Gemini 4, is reported on only ten tasks with one baseline run per task, so it is not directly comparable.
The authors score each system on five axes: First-Attempt Performance, Efficiency, Success, Budget and Improvement. The split is the point. A single endpoint number hides when the gains arrived and how many robot steps they burned.
The headline table
| System | Tasks solved (of 18) | First attempt | Efficiency | Success |
|---|---|---|---|---|
| GPT-6 Astra | 15 | 0.368 | 0.859 | 0.851 |
| GPT-6.1 Sol | 12 | 0.284 | 0.822 | 0.703 |
| Claude Opus 5.5 | 13 | 0.177 | 0.609 | 0.744 |
| Claude Sonnet 5.5 | 11 | 0.124 | 0.695 | 0.701 |
| GPT-6 Sol | 7 | 0.031 | 0.466 | 0.472 |
| GPT-5.6 Sol | 1 | 0.012 | 0.225 | 0.104 |
Astra leads on all five axes in the paper's reading. The more interesting row is Opus 5.5 against GPT-6.1 Sol. Opus has the higher Success score (0.744 versus 0.703) but the lower Efficiency (0.609 versus 0.822), and on the tasks each system solved it saved less of its budget (0.579 versus 0.808). The paper says that gap persists on the ten tasks both systems solve. The two Claude systems reach 90% of their peak gain late, at round 15 (Opus) and 13 (Sonnet), versus round 7 for GPT-6.1 Sol and round 9 for Astra.
Public benchmarks are a weak guide here
Across the four GPT systems, average scores on 17 public benchmarks span 54.1 to 78.1, while the physical profile mean spans 13.9 to 73.4. GPT-5.6 Sol scores above Astra on LiveBench code generation and completion, yet solves one task against Astra's 15. The authors' exploratory read of six systems associates spatial reasoning with a good first attempt and perception with eventual success and recovery, and coding with neither. They are explicit that these are candidate bottlenecks, not causal findings.
Does the improvement stick?
In a six-task ablation on Astra, removing history cut late-round success from 41.7% to 8.3%. With further revision disabled, frozen workspace files still produced success in 25 of 60 trials on unseen layouts, and three cross-task cases reached first success earlier when started from another task's workspace. Early rounds also forecast later ones: fitting Astra's curve to GPT-6.1 Sol's first five rounds predicted the remaining rounds with a mean absolute error of 0.019, against 0.112 for simply holding the last score.
Caveats you should weigh
This is six systems, one improvement chain per task, and baseline repeats that do not estimate variability across whole chains. The ablation covers one system and six tasks; the transfer result is three cases. Public benchmark scores come from differing evaluation settings. Budget and Improvement use task sets that differ per system. The authors themselves call the scaling and bottleneck claims hypotheses. Galbot also authored the benchmark, and it has not been independently reproduced.