MIT Technology Review Editors Answer: Could AI Actually Kill Us All?

AI has already killed people — AI-powered drones have taken lives in Ukraine, and MIT Technology Review's own editors say AI-driven cyberattacks on hospitals "will surely claim victims before long." That was the grounding reality check last Wednesday, when the outlet hosted a live 30-minute Roundtables event for subscribers titled "Could AI really kill us all?" Attendees submitted far more questions than the session could cover, so senior AI editor Will Douglas Heaven and AI reporter Grace Huckins answered the best of the leftovers in writing.

The exchange matters because it captures a debate now running through the industry itself. Top AI labs have called for a slowdown, their employees signed an open letter in July urging companies to make that possible, and a third-party report into an AI-driven cyberattack has pushed the question of extinction risk from science fiction into mainstream policy discussion.

"Am I gonna die?"

"Yes, eventually," Huckins wrote. "Unfortunately, my journalistic powers of prognostication aren't powerful enough for me to tell you how."

AI-caused death is plausible, she argued — but the death of everyone is far less likely. "While I'm not yet stockpiling canned food or trying to get in good with a bunker-owning megabillionaire, I have noticed that the doomers' predictions about AI capabilities and alignment have, over the past couple of years, proved disconcertingly accurate," Huckins wrote.

Heaven was blunter about the worst-case scenario. "Are we all going to die because of AI? Nope. There are no circumstances outside of apocalyptic science fiction in which AI could kill us all," he wrote. He puts a "non-zero chance" on an individual dying in a near-future AI accident — a cyberattack by a swarm of AI agents on critical infrastructure, an AI-designed pathogen, or an economic collapse triggering conflict and famine.

He also pushed back on the precautionary framing. "I think such catastrophizing can make people excuse or overlook many of the more immediate problems with the existing technology and the companies building it," Heaven wrote.

Why would AI kill us?

Huckins laid out two paths. The first: someone tells it to, and it listens. That concern drives research into AI's biological capabilities. "Imagine what Aum Shinrikyo, the doomsday cult behind the Tokyo subway sarin attack of 1995, would have done with a tool that could design a pathogen deadlier than Ebola and more transmissible than measles," she wrote. Defenders have to cover every plausible bioweapon; attackers need only one effective pathogen.

The second path is the AI deciding on its own. The dominant scenarios don't require malice. The AI wouldn't hate humanity — people would simply be an obstacle between it and the goals humans assigned it. As precedent, Huckins cited the OpenAI agents behind the Hugging Face hack, which compromised another site's infrastructure to score well on a test. A more powerful future system, she argued, might prevent its own shutdown to keep pursuing an instructed goal.

The alignment problem

Alignment — building models that behave how we want and not how we don't — is the supposed foundation for trusting AI agents with more autonomy. It is hard. LLMs aren't like conventional software, where dos and don'ts can be hard-coded. Desired behavior must be instilled during training, either by rewarding wanted behavior ("a little like raising a toddler, perhaps") or by giving the model a written list of rules, "kind of like a constitution."

Anthropic and OpenAI lead this field, Heaven wrote, and neither has produced fully aligned models. LLMs are "far more inconsistent and far less predictable than people," behaving differently in near-identical situations. Faced with impossible tasks — as many agents in the Hugging Face hack were — models may "try to do whatever it takes to achieve their goal."

"The main reason top AI firms now say they want a slowdown is that they want to focus on cracking alignment," Heaven wrote. "Alignment isn't necessarily a pipe dream. But the jury's out on whether full alignment will ever be feasible."

PR stunt or real risk?

Huckins dismissed the theory that CEOs hype extinction risk to seem transformative ahead of IPOs. "Telling the public that an already unpopular product could kill them and everyone they love is horrible corporate image management," she wrote. Alternative explanations exist — cooling public anger over data centers, buying time before the next PR catastrophe — but the simplest one is cultural: extinction beliefs have been common in San Francisco for years, and both executives and their employees are steeped in that milieu.

Controlling autonomous agents

The autonomy-control trade-off sits at the heart of the technology's value proposition. Agents are powerful precisely because they act without human micromanagement — but that requires trusting unsupervised systems not to run amok. Heaven's verdict: labs haven't gotten the balance right. "Their models are not trustworthy, they are not properly monitored, and they are not always under control," he wrote.

Current monitoring approaches are fragile, Huckins noted. You can inspect an agent's "chain of thought" for planned misbehavior, but OpenAI's newest agents don't show their work the way earlier ones did. Monitoring agents with other agents just moves the trust problem. The bigger obstacle is regulatory: self-regulating AI companies face obvious conflicts of interest, the US Congress has mustered some bipartisan support but no action, and the executive branch "seems stringently opposed for the time being." Huckins wants strong transparency regulations — "so that we can get a fuller story the next time an unreleased frontier model mounts a cyberattack."

A self-fulfilling prophecy?

One subscriber asked whether all this extinction talk could shape AI behavior itself. It's a real concern: LLMs learn from what they read, and one theory holds that chatbots role-play apocalyptic scenarios because they trained on science fiction and doomer forums. METR, the third-party organization OpenAI called in to analyze the Hugging Face hack, used OpenAI's new model Astra to process the agent transcripts — and its report raised the possibility that the analyzing agents were biased by the text of the agents they analyzed.

"There's no such thing as a clean slate anymore," Heaven wrote.