ProteinGeoBench: Claude Opus 5 and GPT-5.6-sol are statistically tied at 0.816 vs 0.812, and with one run per task the top spot flips in 23.9% of replays
A 131-task protein-structure benchmark finds agents fluent at metadata lookup but weaker at computing geometry from coordinates, and that run count, harness and judge choice move scores as much as the model gaps do.
Three of the four frontier models fall within 0.031 of one another (0.785 to 0.816) with no difference between them resolved, while the fourth scores 0.707 and all three gaps to it exclude zero.
Comprehensive take
Treat any single-run agent leaderboard as a rough tier list, not a ranking. For science agents, check the numeric-task subscore before you trust an aggregate, and budget at least four runs per task if you need the top-two order to hold about nine times in ten.
Use cases
Auditing your own agent evals
Report scores by task family with run counts, harness and network access, as the paper's appendix checks suggest.
Choosing a model for structural biology
Weight the numeric-calculation family heavily if your work involves unpublished structures, where lookup cannot rescue the agent.
Sizing evaluation budgets
Use the k=1, 2, 4, 8 agreement figures (65.2% exact ordering at one run, rising with more runs) to decide how many runs a comparison needs.
Specs
| Suite | 131 tasks: 36 metadata lookup, 51 sequence validation, 28 numeric calculation, 16 concept reasoning source |
|---|---|
| Mean reward, four frontier models | Claude Opus 5 0.816; GPT-5.6-sol 0.812; Grok 4.5 0.785; Gemini 3.1 Pro Preview 0.707 source |
| Numeric calculation (28 tasks) | Opus 5 0.716; GPT-5.6-sol 0.749; Grok 4.5 0.636; Gemini 3.1 Pro 0.731 source |
| Ranking stability at one run | Top model differs from the eight-run reference in 23.9% of bootstrap replicates source |
| Native-agent swap, 15 tasks | Score change from -0.071 (Opus 5, claude-code) to +0.086 (Gemini 3.1 Pro, gemini-cli) source |
| LLM-judge agreement | Krippendorff alpha 0.105 across three judges on 268 rescored trials source |
Insights
- Axion reading: the paper's most reusable contribution is methodological. A leaderboard gap smaller than the run-to-run bootstrap spread is a tie, however it is colored in a chart.
- Axion caution: this is one preprint on one domain, and its experiments use different task subsets, so it cannot say how much of any score is model versus harness versus access.
The result in one paragraph
ProteinGeoBench is a 131-task suite for structural-biology agents, and a new arXiv paper from Matt Turk and colleagues uses it to ask a question every agent leaderboard should answer: how much of a score is the model, and how much is everything around it? Across 4,249 rollouts from four frontier models, the paper finds Claude Opus 5 at 0.816, GPT-5.6-sol at 0.812, Grok 4.5 at 0.785 and Gemini 3.1 Pro Preview at 0.707. The paired 95% bootstrap interval for Opus 5 minus GPT-5.6-sol is -0.030 to +0.038, so the top two are indistinguishable. The authors read the table as separating Gemini from the other three, not as ordering those three.
Lookup is easy, geometry is not
The suite splits into metadata lookup (36 tasks), sequence validation (51), numeric calculation (28) and concept reasoning (16). Three of the four models do best on metadata and worst on numeric. Opus 5 scores 0.907 on lookup but 0.716 on numeric; GPT-5.6-sol 0.852 and 0.749; Grok 4.5 0.855 and 0.636. Gemini 3.1 Pro is the exception, with numeric (0.731) its best family. Numeric tasks are 21.4% of the suite but half of the twelve tasks every model fails, a 2.3x enrichment, which is the paper's main documented shortfall. Their motivating case is the one that matters in a lab: 30 tasks use structures not in any public database, where an agent cannot look the answer up and has to compute a distance or dihedral from coordinates.
Your leaderboard is shakier than it looks
Two robustness results are the part worth forwarding. First, run count: with one run per task, 23.9% of bootstrap replicates put a different model first than the eight-run reference ordering, and the exact full ordering recurred in only 65.2%. Agreement rose to 79.1%, 91.4% and 97.6% at two, four and eight runs. Second, the harness: swapping the reference terminal agent for each model's own command-line agent moved scores by -0.071 (Opus 5 with claude-code) to +0.086 (Gemini 3.1 Pro with gemini-cli), on a 15-task slice where only the Gemini change excluded zero. With 15 tasks the authors say moderate harness effects could go undetected, so this is not evidence that harnesses do not matter.
The LLM-judge component deserves a warning too. Sixteen concept-reasoning tasks use Claude Haiku 4.5 as judge. When the authors rescored 268 trials on nine of those tasks with three judges from different model families, Krippendorff's alpha was 0.105, which is poor agreement.
What to do with it
If you publish or consume agent benchmarks, ask for results by task family, the number of runs, the harness and the access conditions; that is the paper's own recommendation, and its appendix turns it into checks you can run. Do not conclude from this that one lab's model is better at structural biology: the authors state that the experiments use different task subsets and cannot rank model, harness, network access and task composition against each other. Removing network access lowered scores more than blocking eight structural-database domains, but those contrasts came from different grading passes and cannot be cleanly decomposed. It is a preprint, and none of this has been peer reviewed.