OpenAI's GPT-6 Astra hits 87.4% on MultiLangBench, but the COBOL migration eval is already near saturation
OpenAI says its newest models jump from 26.8% to 87.4% on a 20-language benchmark and from 5.4% to 86.6% on COBOL migration. The numbers are real; the sample sizes are not published.
Together, they test two complementary skills: building new software and preserving what existing software does. A rewrite that rounds those values instead can run without errors while producing different financial results. Our latest models show substantial improvements across both evaluations, delivering much higher accuracy at lower cost.
Comprehensive take
Try it for Fortran, OCaml, Visual Basic and Zig work now; for COBOL migration, pilot with your own hidden tests before trusting the 97.9%.
Use cases
Legacy language maintenance
Teams holding Fortran or Visual Basic codebases get the largest measured gains: 21.4% to 83.5% and 27.5% to 89.4% respectively on MultiLangBench with GPT-6 Astra.
COBOL to Java/Rust migration pilots
GPT-6.1 Sol reaches 86.6% at low reasoning effort and 97.9% at higher effort on OpenAI's hidden-test migration eval, with byte-for-byte output parity required.
Cost-sensitive batch coding
GPT-6 Luna's estimated $0.77 per sample versus $3.48 for GPT-5.6 Sol makes it the candidate for high-volume, lower-accuracy-tolerant jobs, at 40.3% average pass rate.
Specs
| MultiLangBench average (GPT-6 Astra) | 87.4% source |
|---|---|
| MultiLangBench average (GPT-5.6 Sol baseline) | 26.8% source |
| COBOL migration (GPT-6.1 Sol, low effort) | 86.6% source |
| COBOL migration (GPT-6.1 Sol, higher effort) | 97.9% source |
| Est. cost per sample, Luna vs 5.6 Sol | $0.77 vs $3.48 source |
| Languages in MultiLangBench | 20 source |
Insights
- COBOL remains the hardest MultiLangBench language at 72.1% even for the top model.
- The migration eval is near saturation at 97.9%, so it will not discriminate the next generation.
- No task counts, variance or confidence intervals are published; per-language results are on GitHub for replication.
Axion Scorecard
Overall: 7.0
87.4% average across 20 languages, up from 26.8%; 97.9% on COBOL migration. source
Eval harness and per-language results on GitHub; models themselves are closed. source
Not yet scored — no cited data
Luna estimated at $0.77/sample vs $3.48; 6.1 Sol at 61% of prior cost on migration. source
States caveats on review, but omits task counts, variance and confidence intervals. source
Two new evals targeting an under-measured problem: low-resource languages and behaviour-preserving migration. source
What OpenAI actually published
OpenAI's 19 October post introduces two evaluations and reports four models against them. The first, MultiLangBench, adapts tests from SQLite's test suite into 20 languages: C, C#, C++, COBOL, Common Lisp, Fortran, Go, Java, JavaScript, Lean, OCaml, PHP, PowerShell, Python, Ruby, Rust, Scala, Swift, Visual Basic and Zig. The candidate program receives an initial database and an SQL query, produces a modified database, and a reference SQLite implementation checks the result. Scoring is test pass rate plus API cost over an extended session.
The second is a COBOL migration evaluation with hidden tests. Models migrate COBOL programs to C++, Java, Kotlin, Python or Rust, and a migration only counts if every hidden test passes with byte-for-byte parity on observable outputs.
The headline numbers, exactly as stated
| Evaluation | Model | Score | Baseline (GPT-5.6 Sol) |
|---|---|---|---|
| MultiLangBench, max reasoning | GPT-6 Astra | 87.4% avg | 26.8% |
| MultiLangBench, max reasoning | GPT-6 Luna | 40.3% avg | 26.8% |
| MultiLangBench COBOL | GPT-6 Astra | 72.1% | 10.9% |
| MultiLangBench OCaml | GPT-6 Astra | 85.8% | 25.6% |
| MultiLangBench Fortran | GPT-6 Astra | 83.5% | 21.4% |
| MultiLangBench Visual Basic | GPT-6 Astra | 89.4% | 27.5% |
| COBOL migration, low reasoning | GPT-6.1 Sol | 86.6% | 5.4% |
| COBOL migration, higher reasoning | GPT-6.1 Sol | 97.9% | not stated |
On cost, OpenAI estimates GPT-6 Luna at $0.77 per MultiLangBench sample against $3.48 for GPT-5.6 Sol, roughly 78% lower, and says GPT-6.1 Sol runs the COBOL migration eval at 61% of GPT-5.6 Sol's API cost.
What is new versus the press release framing
Three things stand out when you read past the headline.
First, COBOL is still the weakest language. Astra clears 83% in 19 of 20 languages but lands at 72.1% on COBOL. The gap between COBOL and everything else narrowed, but it did not close, and the post does not say what the remaining failures look like.
Second, the COBOL migration eval is nearly saturated. At 97.9% with higher reasoning effort, there is little headroom left to separate the next model from this one. Practitioners should treat that number as 'the eval is done', not as 'migration is solved'.
Third, the two evals are reported at different reasoning effort settings. MultiLangBench is quoted at maximum effort; the 86.6% migration number is at low effort. That is a fair way to show the cost story, but it means the two tables are not directly comparable to each other.
Caveats OpenAI states, and ones it does not
OpenAI itself says Luna's cost figures are estimates, that migrated production code still needs expert review against business rules and dependencies, and that the models still fall short in unspecified areas.
What the post does not report: number of tasks per language, variance across runs, confidence intervals, or how many COBOL programs are in the migration set. Without task counts, a 2-point difference between languages is uninterpretable. The MultiLangBench harness and per-language results are released on GitHub, so this is checkable by anyone willing to spend the API budget.
Axion verdict
If you maintain Fortran, OCaml or Visual Basic, the jump from roughly a quarter of tests passing to over 83% is the kind of step change that justifies a pilot this quarter. If you are planning a COBOL migration, the 97.9% figure is encouraging but the post's own warning applies: byte-for-byte parity on a hidden test set is not the same as parity on your production batch jobs. Run the released MultiLangBench harness against your own stack before switching anything.