OpenAI Says GPT-6 Astra Passes 87.4% of Legacy-Language Coding Tests

OpenAI says its newest models can now work effectively in COBOL, Fortran and Visual Basic — programming languages with a rapidly shrinking pool of human maintainers — and it has benchmark numbers to back the claim. GPT-6 Astra achieved an average test pass rate of 87.4 percent across 20 languages on the company's new MultiLangBench evaluation suite, compared with 26.8 percent for GPT-5.6 Sol.

The figure matters because of who OpenAI is addressing: banks. According to the company, 40 percent of the world's banks still rely on COBOL systems, and migrating those systems to newer languages would cost each institution millions to hundreds of millions of dollars, plus considerable time. The programmers who maintain COBOL are aging out of the workforce, OpenAI notes, which means banks still running the language will have to migrate within the next few years.

By Marcus Bennett

Why is COBOL migration suddenly an AI story?

COBOL (Common Business-Oriented Language, a programming language dating to 1959 and still running core banking and insurance systems) presents a specific economic problem. The code works. The people who understand it do not remain. OpenAI frames its model improvements as a direct answer to that labor gap.

The company said its work on improving COBOL capabilities is guided by developers and enterprise customers who submitted feedback. That feedback revealed a concrete weakness: coding models struggle when working with old languages. Modern benchmarks overwhelmingly test performance on current languages such as Python and JavaScript, leaving legacy-language capability undermeasured.

To close that measurement gap, OpenAI built MultiLangBench, an evaluation suite that asks models to create SQLite-compatible database engines in a specified programming language. The design choice is deliberate. Building a working database engine in a target language forces a model to demonstrate real fluency in that language's syntax, memory behavior and standard libraries rather than pattern-matching surface syntax.

What do the benchmark numbers show?

The MultiLangBench results, as reported by OpenAI:

• GPT-6 Astra: 87.4 percent average test pass rate across 20 languages
• GPT-5.6 Sol: 26.8 percent average pass rate across the same suite
• COBOL specifically: 72 percent pass rate for GPT-6 Astra — the hardest language to crack in the suite

The gap between Astra's overall average (87.4 percent) and its COBOL-specific rate (72 percent) is the honest number in the dataset. COBOL remains the most difficult language in the evaluation even for the strongest model. A more than 60-point improvement over GPT-5.6 Sol's overall average, however, indicates that legacy-language performance moved from effectively unusable to plausibly production-adjacent within a single model generation.

OpenAI did not break down Fortran or Visual Basic pass rates individually in its post. The company also did not publish the full per-language distribution across the 20 languages tested.

Can the models actually preserve what COBOL programs do?

Pass rate on a synthetic benchmark is one thing. Bank migration is another. OpenAI therefore evaluated its models on a second, more demanding task: migrating COBOL programs to C++, Java, Kotlin, Python or Rust while preserving what the programs do.

Behavior preservation is the hard constraint. For OpenAI's models to effectively help banking institutions migrate, they must preserve COBOL programs' behaviors — such as how a program rounds numbers with excess decimal digits and how it handles missing records. These are not edge cases in banking software. Rounding behavior determines whether ledger totals reconcile. Missing-record handling determines whether a payment fails gracefully or corrupts state.

On this migration-accuracy evaluation, OpenAI reported:

• GPT-6.1 Sol at high reasoning effort: 97.9 percent accuracy, the highest of any model tested
• GPT-6.1 Sol at low reasoning effort: 86.6 percent accuracy

The spread between the two reasoning-effort settings — 11.3 percentage points — quantifies the computational cost of behavior fidelity. Institutions considering these tools will need to weigh that trade-off directly: cheaper, faster inference against a measurable accuracy drop on the task where accuracy matters most.

What does OpenAI concede about the limits?

The company's own framing includes a notable admission. In its post, OpenAI states that a migration is only successful if "all tests pass." For programs that handle banking tasks, that bar follows from the domain: banking software requires high accuracy by default, not as an aspiration.

OpenAI also quoted its own guidance verbatim:

"Behavior-preserving capabilities like these are useful for teams maintaining and modernizing older systems, though applying it to production software still requires expert review and testing against the system's business rules and dependencies."

That sentence carries weight because it comes from the vendor. OpenAI is not positioning its models as autonomous migration engines. It is positioning them as assistive tooling for teams, with expert human review and business-rule testing as non-negotiable steps before production use. The 97.9 percent figure, read against the company's "all tests pass" standard, makes the gap explicit: near-perfect is not perfect, and banking systems tolerate neither silent rounding changes nor divergent missing-record handling.

Why does this matter for the market?

The commercial logic is straightforward. If 40 percent of the world's banks still run COBOL, as OpenAI claims, the addressable migration problem spans thousands of institutions, each facing costs in the millions to hundreds of millions of dollars. The workforce clock is the forcing function: COBOL programmers are retiring, and OpenAI's assessment that banks "will have to migrate within the next few years" converts a deferred IT problem into a near-term procurement decision.

Enterprise modernization is also a proven willingness-to-pay category. Vendors that can credibly reduce migration cost, timeline and staffing risk stand to capture spending that would otherwise go to consulting firms and specialized legacy-system integrators. OpenAI's decision to publish benchmark numbers, name the specific languages and quote its own limitations suggests it is courting exactly those enterprise buyers, who will demand evidence rather than demos.

The MultiLangBench release functions as that evidence, and as an invitation. A public, repeatable evaluation suite for legacy languages gives customers a way to score competing models on the same task — and gives OpenAI a leaderboard it currently tops with 87.4 percent.

What happens next?

The open question is whether assistive AI migration can compress timelines enough to beat the retirement curve of the remaining COBOL workforce. OpenAI's own numbers frame the challenge: 72 percent on COBOL construction, 97.9 percent on behavior-preserving migration at high reasoning effort, and an explicit requirement for expert review before production — figures that support accelerated, human-supervised modernization rather than full automation. For banks still holding COBOL systems in the next few years, the decision is now less about whether to migrate and more about which tools, and how much review, the migration will require.