OpenAI says its newest models can translate COBOL — experts aren't buying it yet

GPT-6.1 Sol preserved the behavior of aging COBOL programs with 98 percent accuracy in OpenAI's own migration tests, the company announced Monday — and that number is either a turning point or a mirage, depending on which software researcher you ask.

OpenAI touted "substantial improvements" in its latest models' ability to migrate software written in older programming languages into modern code, saying the gains could help organizations with "maintaining and modernizing older systems." The stakes are enormous. COBOL, born in 1959, still underpins a staggering share of the world's critical infrastructure, and nobody has managed to retire it in nearly seven decades.

Why does COBOL still matter in 2025?

The numbers explain the urgency. According to IBM, COBOL forms the foundation of more than 40 percent of online banking systems, supports 80 percent of in-person credit card transactions, and handles 95 percent of ATM transactions. It also runs government systems across the United States — Social Security, the IRS, and services operated by most states. One rough estimate puts the annual cost of COBOL's inefficiencies to the US GDP at around $105 billion.

Yet replacing it is fiendishly hard. These systems are often too important to risk breaking, and many of the programmers who understood them have retired or died. The same applies to other legacy languages. Fortran, for instance, still powers scientific computing, weather forecasting, atmospheric and ocean studies, and codes critical to nuclear security.

For all their gains in modern software development, AI coding tools have struggled to make headwind here. OpenAI's latest results suggest that may be starting to change — but only starting.

What did OpenAI actually test?

In a blog post published Monday, OpenAI reported promising results migrating COBOL into C++, Java, Kotlin, Python, and Rust without changing what the original software does. The company acknowledged that "a migration is considered successful only if all tests pass" — a crucial bar for any organization hoping to ditch its legacy code.

The benchmark numbers move fast across model generations:

• GPT-6.1 Sol: 87 percent behavior preservation accuracy at low reasoning effort, climbing to 98 percent at higher reasoning
• GPT-6 Astra: around 86 percent at low reasoning effort
• GPT-6 Sol: 21 percent at low reasoning effort
• GPT-5.6 Sol: a mere 5 percent at low reasoning effort

Reasoning effort refers to how many computational steps a model spends analyzing a problem before producing an answer. The jump from 5 percent to 98 percent across three generations is the story OpenAI wants told.

"These evaluation results demonstrate progress toward more accurate, affordable code migration for teams modernizing the software their businesses depend on," OpenAI wrote. The company also reported improvements in its models' ability to build database software in various programming languages, including COBOL.

OpenAI spokesperson Oscar Haines acknowledged in an email to The Verge that legacy code "is still far from a solved problem." He said the company is listening to feedback from developers and enterprises, and exploring ways to build better tests and continue improving its models' understanding of programming languages.

What is Anthropic doing with COBOL?

OpenAI isn't alone in chasing the legacy-code market. In February, Anthropic published a step-by-step "code modernization playbook" and pitched its coding agent, Claude Code, as a way to migrate COBOL into modern languages like Java or Python.

Anthropic offered no performance benchmarks showing how well Claude Code performs such migrations. It still declared that "the economics of COBOL modernization have shifted." The market noticed. Shares in IBM, which derives part of its revenue helping companies maintain legacy systems, nosedived following the announcement. Anthropic did not provide comment in time for publication.

Why are experts skeptical?

Researchers who spoke to The Verge agreed that AI tools have made real strides on legacy code in recent years. But they cautioned that OpenAI's advances remain a very long way from making it safe to replace critical systems built on it.

"Rewriting code is not the same as replacing a system," said Vadim Zaytsev, an associate professor of software evolution at the University of Twente in the Netherlands. Size alone is deceptive. Zaytsev said the largest system he'd seen had 250 million lines of COBOL — and it was "definitely not the world's largest." Code may be written in multiple languages. Data may be stored in various formats. Technical integrations with other systems, and the expectations of the people and organizations running them, all complicate the picture.

That complexity means "AI is not playing a major role at the moment in replacing or migrating legacy COBOL systems," Zaytsev said.

Arie van Deursen, a software engineering professor at Delft University of Technology, went further. He described OpenAI's blog as "superficial," noting it offered few details about the migration task, while its language benchmark tested models on a relatively simple, well-defined task involving systems similar to those the models would have encountered in training. He also stressed the gap between maintaining an aging system and actually modernizing it. "Maintaining is of course a lot easier than modernizing," van Deursen said, calling the tasks OpenAI used to assess its models "lightyears" away from real modernization work.

"The evaluations shown by OpenAI don't address the key difficulty: Breaking up a complex legacy system into smaller units that can be replaced individually," van Deursen said.

What would make AI migration safe?

The hardest part isn't even reading the old code. "The main challenge is indeed not understanding how the system works," said Alastair Donaldson, a professor of programming languages at Imperial College London. That lack of understanding is a "very common" problem for legacy systems, he said, and very few people still know how to program in languages like COBOL. As for new projects, Donaldson was blunt: "nobody in their right mind would use COBOL for a new development project."

Robust testing is the path forward, Donaldson said. One approach: run the same workflows through the old and new systems and check they behave identically. AI could also document its code translation, letting humans familiar with the legacy language review the work.

Formal verification — mathematical techniques that computationally prove code behaves correctly — is another option, said Jacopo Mauro, a mathematics and computer science professor at the University of Southern Denmark. Such methods already verify AI-generated mathematical results, but applying them to legacy code would require a "huge amount of work," Mauro said. The payoff would be automatic checking of translated code, which he argued beats manual review: "having this will be better than having a human check that the translation is correct." Until then, he said, "human verification remains essential."

For systems as critical as those running on COBOL, near-perfect isn't good enough. "In many cases even minor differences in system behaviour before and after migration are unacceptable," Zaytsev said. AI can deliver savings and speed, but partial automation doesn't close the gap.

"Suppose you have a magic AI wand that handles 99 percent of your code," Zaytsev said. "What do you do with the remaining 1 percent? That 1 percent can be a million lines of code, spread all over the place and subtly failing in more ways you can imagine within your lifetime."

The trajectory is real — from 5 percent to 98 percent in three model generations — but until AI can decompose, verify, and safely replace entire systems rather than translate isolated programs, the world's ATM networks and tax agencies will keep running on code written before most developers were born.