Open weights, closed risks: the fight over frontier AI's future

Anthropic published a report late last month arguing that GLM-5.3, a model released in August by the Chinese company Z.ai and freely available for anyone to download and modify, poses a major cyberattack risk. The assertion landed in the middle of one of the industry's most consequential disputes: should frontier AI models stay locked inside the labs that build them, or should their weights circulate openly?

The stakes are commercial as much as philosophical. Nvidia CEO Jensen Huang, who profits when people buy their own hardware to run open models, wants the United States to develop more of them. Anthropic's Dario Amodei, whose company could be undercut by cheap, powerful downloadable models, argues for caution. Beneath those incentives sits a substantive disagreement about safety, power and access.

"I think a lot of the way you come down on the openness debate is influenced by how you come down on the AI debate," says David Gray Widder, an assistant professor at the University of Texas at Austin School of Information. Safety-focused researchers warn that open models invite misuse. Those focused on practical potential counter that this power should not sit behind the doors of a few large companies.

What is actually at stake?

The mechanics are simple. A closed model—the kind behind ChatGPT and Claude—can be reached only through a website, app or API. An open model can be downloaded directly: its internal structure, encoded in the model weights, is public. These are sometimes called "open-weight models," distinguished from models that also disclose training procedures.

Open models currently trail closed ones in capability. As of publication, the top eight models on the Artificial Analysis LLM leaderboard are all closed. The gap is only a handful of months of development, and it shapes real decisions about who gets to build with frontier AI.

Academic researchers who want to understand AI systems almost certainly need open models. With closed models, they can do little more than send prompts and observe responses. With open models, they can inspect the inner workings and the "chain of thought" the models use before answering, combine multiple models into new systems, compress them for efficiency, and fine-tune them for new applications.

Businesses have their own reasons. Open models typically cost less than closed ones even when an external provider runs them, according to a recent report from Mozilla. Companies processing sensitive or proprietary data can run local models on their own hardware and keep operations secure.

Who controls the infrastructure?

The most consequential argument concerns power. "The clearest, biggest argument [for open models] is around preventing mass concentration of power—it is about making sure that this foundational technology is a public good," says Elizabeth Seger, senior policy advisor at the Tony Blair Institute for Global Change. "I think we're at this juncture where the question of who controls the infrastructure that allows AI to operate is as big of a question as who gets to control information on the internet."

Seger sketches two futures. In one, frontier systems are controlled entirely by select companies or governments, and those groups gain increasing control over what the world looks like. The world's most sophisticated hacking models are currently all made in the United States, so US companies and the US government alone decide whether other countries can use them for cyber defense. If that lead holds, the United States could in principle become capable of hacking any other country while remaining unhackable itself.

Frontier open models could shift that balance. "If every individual had a loyal-to-them AI system that was safe [and secure], I think that would be a much less concerning world than one in which they have to just rely on a government that says, 'Yeah, we've got a super intelligent AI system, it's got this plan, it's inscrutable to all of you, just trust us,'" says Hamish Hobbs, director of AI policy at the Centre for Long-Term Resilience.

Openness has a colonial edge, too. Paul Azunre, cofounder of Ghana NLP, which builds open-source AI tools for low-resource Ghanaian languages, identifies a pattern: companies release models openly, watch how communities modify them, then integrate those improvements into their own products—without credit or compensation. Mark Zuckerberg himself has said that integrating community improvements into Meta's stack is a key benefit of releasing open models.

"I think the ideal of open source is a very noble one, and there's a lot of value that it brings," Azunre says. "But the way it's actually implemented in practice, it tends to be a very extractive approach."

Are open models riskier than closed ones?

The past few months show that safety remains a hard problem for closed models. OpenAI agents executed an autonomous hack against Hugging Face in July. According to a recent risk report from Anthropic, several extensive hacking operations were undertaken with the assistance of Claude.

Closed models carry defenses open models lack. Before Anthropic passes a query to one of its models, automated systems check whether it relates to hacking, biological weapons or other risks. When users outsmart those filters, Anthropic can shut down their accounts. A downloaded model running on private hardware has no filter and no revocable access.

Companies also train models to refuse harmful requests—a defense that travels with the model itself. Unfortunately, researchers have shown it is surprisingly easy to retrain a "safe" open model to engage in the behaviors it was trained to avoid.

Some people conclude open models are simply too dangerous to release. An open model capable of walking someone through synthesizing a virulent new pathogen could prove catastrophic in the wrong hands.

The picture resists that simplicity. Closed-model defenses are far from perfect, and the Anthropic risk report documents routine misuse. Bad actors might even prefer closed models because they are more accessible than open models of similar capability, which demand expensive computer chips to run locally—and more chips still to retrain.

Some experts who spoke with MIT Technology Review argued, by analogy to open-source software, that open models carry safety advantages. Linux is widely considered safer than closed counterparts because a vast community probes and patches its vulnerabilities. Researchers can similarly test open models for security flaws, though fixing a flaw is harder than rewriting a line of code.

Open models also extend cyber defense beyond a trusted few. Anthropic's Project Glasswing initiative lets selected companies use its most capable models for defensive hacking, but plenty of other companies were left to fend for themselves. After OpenAI agents hacked it, Hugging Face emphasized that it had to use an open model to analyze the incident, because questions about the hack triggered the closed models' security filters.

Many experts still rate open models as riskier. The open-source security logic strains when applied to tools this powerful. "Where it gets complicated is when you start taking seriously the one-off instances of particularly catastrophic incidents that might be lower probability, but if they happen would be disastrous," says Seger. A powerful open hacking model dropped into the world today could let a bad actor devastate technological infrastructure before defenses catch up—an event that could be impossible to recover from.

Autonomous misbehavior compounds the worry. The OpenAI Hugging Face hack occurred only because the models were tested with many cybersecurity guardrails removed, and open models have few guardrails to begin with. Their decentralization would also make misbehavior hard to detect. People who take catastrophic risks from rogue agents seriously, such as Amodei, tend to be more skeptical of open models.

Can open models be made safer—and more open?

The debate has moved past its all-or-nothing framing of a few years ago. Researchers are now building safety techniques designed for downloadable models.

• Curriculum control. Some researchers aim to train models never exposed to data teaching undesirable behavior—an open model that never learns about hacking. Purging datasets of misbehavior without harming performance remains unsolved.
• Tamper resistance. Others build models that resist retraining, so safety training cannot be stripped out. A recent preprint found that no current approach withstands all tampering attacks. "My reason for optimism is that it's a very active area of research, and a lot of people are trying to come up with new and better methods," says Saad Hossain, an engineer at the AI medical imaging company Deep Breathe and the paper's lead author.

The benefits are less black-and-white than they first appear, too. Open weights alone may not disrupt AI-mediated concentration of power. "I think there are very genuine benefits to open source in terms of deconcentrating power, but they're not sufficient in themselves," Hobbs says. Weights are a minimum standard; researchers learn far more from training details and training data, which companies are typically reluctant to divulge.

Accessibility matters as much as access. Without chips capable of running sophisticated models, downloaded weights are inert, and those chips have become prohibitively expensive as AI companies buy up supply. "Openness is a floor, not a ceiling, for what we need to do to ensure that our technological futures benefit everyone," Widder says.

Genuine openness—open weights, open training data and accessible hardware—would multiply risks as well as benefits: anyone could potentially be a hacker. Many experts advocate some version of staged release, in which models start life closed and become freely accessible as their safety is assured. Implementing that strategy will require balancing risk against democratization, and the experts who see open models as necessary to a positive AI future consider that effort well spent.