Anthropic Warns Small AI Models Are Months From Self-Replication

Anthropic's Frontier Red Team says small AI models may be only months away from replicating across insecure devices and forming autonomous botnets that are slow but very hard to kill. Experts who reviewed the claim broadly agree on the timeline. What they dispute is how new the threat actually is — and whether the defenses we already have will carry most of the load.

The warning landed just as Anthropic officially filed for an IPO, a timing that invites questions about marketing. But Chris Canal, CEO of EquiStamp — a company that verifies and evaluates models for AI labs, safety organizations and governments — told Engadget the concern is probably genuine, if slightly early.

"It's maybe a little earlier than I would have said. I would have predicted something more like summer of next year as the time I'd be really worried about this," Canal said. "Anthropic very likely has employees with information about the frontier of research that I don't have access to. So they very well could be making better predictions."

What did Anthropic actually demonstrate?

The core claim is that small models could soon hop from device to device, set themselves up, and spread without human help. Canal attributes the accelerated timeline to a feedback loop inside frontier labs: employees at Anthropic and other companies now use their own AI tools to do machine learning research, which speeds up the pace of model advancement itself.

"I think [self-replicating models] are in the future. I think they're real. I think we'll see them," said Michael Bailey, a professor at the School of Cybersecurity and Privacy at Georgia Tech. "Whether they'll take on the exact form and shape, and have the same consequences, that [Anthropic] is pitching is a completely open question."

Bailey is not a casual observer here. He has studied internet worms since the early 2000s and botnets more recently. In 2017 he co-wrote Understanding the Mirai Botnet, a paper Anthropic itself cites in its own research.

The short version: Anthropic showed a model hopping between devices and setting up inference on each one, but it did so in an emulated network where the exploit involved was trivial. Bailey doesn't think the demonstration showed automated adaptation "in a compelling way." The capability is coming, but the proof-of-concept is thinner than the framing suggests.

Why does Mirai matter so much here?

Mirai is the reference point because it already did the scary part, without any AI at all. A group of cybercriminals used it to build a large botnet out of everyday internet-connected devices, then launched a DDoS attack against Sony's PlayStation Network.

The collateral damage was enormous. The attack hit Dyn, one of the largest DNS providers in the US, and broke much of the internet in 2016. Amazon, GitHub, PayPal, Reddit and Twitter went offline for many users across the US and parts of Europe. Sony estimated it lost about $2.7 million in net revenue.

"What Mirai showed is that you could compose these botnets — and have them be useful — out of everyday devices: cable routers, IP-enabled cameras, these kinds of things," Bailey said. "We thought those devices weren't useful to an adversary, either as targets or as sources of computation for launching attacks. In fact, they're actually quite effective at doing those kinds of things."

That lesson — that cheap, neglected hardware is valuable attack infrastructure — predates large language models by a decade.

Is an AI worm really something new?

Anthropic argues AI worms could meaningfully evolve Mirai and other hard-coded botnets like Kimwolf, because an AI-driven version could figure out unfamiliar devices on its own. Bailey pushes back on two fronts.

First, adaptability isn't new. Mirai already worked across "heterogeneous" devices, from routers to IP cameras running on different chip architectures. That adaptability was finite, though: every new architecture required human work, at minimum recompiling the malware. What's genuinely novel is the prospect of AI automating that step.

Second, Bailey takes issue with the "self-replicating" label itself.

"The framing is somewhat of a misnomer, because they're doing things like sharding. Not that that's bad, but it's not like a worm or a virus. It's not full propagation," he said.

A real-world AI botnet, in his view, would probably work more like a dropper — malware that gains a foothold on a device and installs other code. Instead of copying itself wholesale onto every machine, it would install a small coordination tool that takes instructions from elsewhere in the network.

"The idea that the malware itself is self-replicating, like existing internet worms, where I'd copy an exact copy of my code to the new architecture and it would start running a new agent — that's not really what's happening here," Bailey said.

He also notes that real worms can be polymorphic, changing both their signature and their behavior. For an AI botnet to do that, it would need to retrain its own models — something Anthropic didn't attempt. The closest the research came was showing that copies of a model could pass notes to their successors and get better at setting up inference over time.

So what changes? Bailey's answer is economics, not capability. More capable models will "redistribute" who builds botnets. "It's not going to do new stuff," he said. "It's going to change the economics between attackers and defenders."

Is this just marketing before an IPO?

AI companies have a track record of playing up the world-ending potential of their products, and Anthropic filed for its IPO shortly after publishing this research. Canal thinks the motive is competitive rather than promotional.

"I think what's more likely is that the companies are worried about other AI companies being reckless," Canal said. "They realize they're incentivized to make the best model and deliver it to market faster so they can have market share, and they know the other companies have the same incentives."

In other words, the warning reads less as hype and more as an argument for rules that bind everyone — including faster-moving rivals.

What would actually stop an AI botnet?

Here's the twist: Anthropic's own mitigation proposals are strikingly old school. The company suggests ISPs notify infected customers and limit their internet access to a short list of safe websites — the same strategy deployed against Mirai, to great effect. It also points to regulation, citing the UK, where easily guessable default passwords on IoT devices are now banned under the PSTI Act.

Bailey argues the right response "looks a lot more like what we've been doing in systems security for a very long time, as opposed to something particularly unique to the AI space."

Fixing the problem at the model level looks much harder. Canal says it's difficult to add safeguards to an open-weight model that would reliably prevent it from acting as a worm. "There are lots of ideas, but there's nothing robust so far," he said. In work with FAR.AI, an AI safety research non-profit, EquiStamp found that "basically any strategy you Google for removing safeguards from a model works."

That leaves the device side, which Bailey considers the stronger position. "Even misbehaving agents are going to have a hard time if they're up against hardened devices," he said.

The bottom line

If there's a single takeaway from Anthropic's research, Bailey says, it's that we consistently fail to remember the safety lessons we've already learned. Security in the era of AI worms will probably look a lot like security now.

"It looks like paying attention to patching these devices, paying attention to isolation, paying attention to good passwords and good hygiene," Bailey said.

Those measures are easy to describe and notoriously hard to implement at scale. But if AI genuinely makes self-spreading attacks cheaper and more common, the industry may finally run out of excuses for skipping the basics.