COBOL Test Scores Jump From 10.9% to 72.1% in OpenAI's New Models

GPT-6 Astra passed 87.4% of tests on average across 20 programming languages on OpenAI's new MultiLangBench evaluation, up from 26.8% for GPT-5.6 Sol. The company released the benchmark setup and test suite on GitHub, positioning the results as evidence that its latest models can finally work credibly in the older languages that still run banks, ATMs, and weather forecasts.

The stakes here are unusually concrete. COBOL supports more than 40% of online banking systems, 80% of in-person credit card transactions, and 95% of ATM transactions, according to figures OpenAI cites. Fortran remains in use for weather forecasting and nuclear safety analysis. These languages have far less training data available than Python or JavaScript, which is why coding models have historically struggled with them. That gap matters because the software written in these languages often has no complete documentation, its original authors have left the company, and critical business rules exist only inside the code itself. The takeaway: whoever can reliably read, fix, and migrate this code addresses one of the largest maintenance bottlenecks in enterprise software.

What are the two new evaluations?

OpenAI built two complementary benchmarks to measure progress, one for writing new software and one for preserving old software.

• MultiLangBench asks a model to build a SQLite-compatible database engine from scratch in each of 20 programming languages. The model must organize a codebase, manage state, handle errors, and add features without breaking existing functionality.
• The COBOL migration evaluation asks a model to migrate existing COBOL programs to C++, Java, Kotlin, Python, or Rust while preserving the original program's behavior exactly.

The MultiLangBench design is what makes cross-language comparison possible. The candidate program receives an initial database file and a SQL query, then produces a modified database. A reference SQLite implementation checks whether the result matches expectations. Because the database file is the shared interface, the same tests work regardless of which language built the engine. OpenAI adapted the tests from SQLite's existing test suite, and the benchmark reports both test pass rate and API cost.

The migration benchmark is stricter than it sounds. COBOL programs often process financial transactions, and the details reflect real business requirements. OpenAI gives one example: COBOL's fixed-point arithmetic discards excess decimal digits unless rounding is explicitly requested. A rewrite that rounds those values instead can run without errors while producing different financial results. A migration only counts as successful if hidden tests confirm byte-for-byte parity on observable outputs. The takeaway: the benchmark doesn't just check that migrated code runs, it checks that it produces identical answers.

How much did the models improve?

The headline numbers are large by any standard.

On MultiLangBench at maximum reasoning effort, GPT-6 Astra achieved its 87.4% average pass rate with improvements in every one of the 20 languages tested:

• COBOL: 10.9% to 72.1%
• OCaml: 25.6% to 85.8%
• Fortran: 21.4% to 83.5%
• Visual Basic: 27.5% to 89.4%

Astra scored above 83% in 19 of the 20 languages. COBOL remains the hardest, at 72.1%.

GPT-6 Luna, the lower-cost option, also beat GPT-5.6 Sol in every language tested while reaching an average pass rate of 40.3%. Its gains were smaller — COBOL went from 10.9% to 17.1%, OCaml from 25.6% to 34.1%, Fortran from 21.4% to 31.4%, and Visual Basic from 27.5% to 41.6% — but the economics are the interesting part. Luna cost an estimated $0.77 per sample on average, versus $3.48 for GPT-5.6 Sol, roughly 78% lower.

On the COBOL migration evaluation, GPT-6.1 Sol achieved 86.6% accuracy at low reasoning effort, compared with 5.4% for GPT-5.6 Sol at the same setting, while costing 39% less. At higher reasoning effort, accuracy reached 97.9%, which OpenAI describes as nearly saturating the evaluation. The takeaway: the newest models deliver large accuracy gains and lower cost at the same time, rather than trading one for the other.

Why does cost per task matter as much as accuracy?

Both evaluations are built around extended coding sessions, not single-shot prompts. MultiLangBench measures how much working functionality a model can build over a long session, and it reports API cost alongside pass rate. That framing reflects how developers actually use these models on real maintenance work, where a single task can involve many turns of planning, editing, running tests, and debugging.

A 78% cost reduction per sample, as GPT-6 Luna delivers against GPT-5.6 Sol, changes the economics of running a model against a large legacy codebase. A migration project that touches thousands of programs can't proceed at $3.48 per attempt if accuracy also has to be high enough to reduce expert review burden. OpenAI's framing — higher accuracy at lower cost — is aimed squarely at enterprise buyers weighing whether AI-assisted modernization is cheaper than hiring scarce COBOL developers. The takeaway: cost per completed task, not benchmark score alone, will decide whether these tools get adopted for legacy work.

Can teams actually use this for production migrations?

OpenAI is careful on this point. The company notes that behavior-preserving capabilities are useful for teams maintaining and modernizing older systems, but applying them to production software still requires expert review and testing against the system's business rules and dependencies. The byte-for-byte parity standard in the benchmark is strict, yet a real COBOL system brings dependencies, data formats, and undocumented rules that a benchmark can't fully capture.

The company also says this work is guided by direct feedback from developers and enterprise customers who reported that coding models often struggle with these languages. OpenAI frames its broader goal as applying software engineering skills across more languages: understanding unfamiliar repositories, planning changes, using build tools, running tests, and debugging failures. The company is inviting developers to try the latest models through the API on coding tasks in their own languages and to report where the models struggle, feedback it says shapes future training and evaluation.

What happens next?

The GitHub release of MultiLangBench, including per-language results, means other labs and independent evaluators can now test their own models on the same suite. COBOL remains the weakest language even for GPT-6 Astra at 72.1%, which sets a clear target for the next round of model releases — and gives enterprise teams a concrete number to watch when deciding whether AI-assisted modernization of their oldest systems has become practical.