Khanna Bill Would Ban Self-Improving AI Until Federal Safeguards Exist

Rep. Ro Khanna, a Silicon Valley Democrat, will introduce legislation that would ban recursively self-improving AI until the federal government builds safeguards and approves such systems. The bill, dubbed the "Human Control Over AI Act," pairs that prohibition with strict liability standards and a sweeping new oversight agency — the most aggressive regulatory architecture proposed in Congress to date for frontier AI.

Khanna shared a summary of the bill exclusively with CNBC and announced it amid mounting pressure on Washington to act. Top executives have warned that AI could slip out of their control, yet Congress has passed essentially nothing to regulate a technology that now underpins large parts of the economy and, in the lawmakers' framing, could inflict catastrophic damage if it fails.

"There's actually a civilizational extinction risk," Khanna said Monday in an interview with CNBC. "There's a safety risk of loss of control, and then there's a misuse risk, and we need to take both seriously."

What the bill would actually do

The core prohibition targets two capabilities that safety researchers have flagged for years: models that recursively self-improve and models that autonomously modify their own core objectives, containment or shutdown controls. Under the bill, developers could not build or deploy such systems until federal guardrails exist and a designated agency signs off on the activity.

That agency would be new. It would focus specifically on frontier models from OpenAI, Anthropic, Google DeepMind and xAI — the four labs at the center of rapidly escalating safety concerns. Its mandate would include:

• Safety regulations governing frontier model development and deployment.
• A licensing system that approves both model training and deployment.
• Frontier AI model audits and security standards for continuous testing and effective human control.
• Embedded independent auditors at every frontier lab, reporting directly to the agency rather than to the companies they audit.
• Containment standards covering sandbox testing environments; air gaps that isolate testing environments from the internet; kill switches; and controls preventing models from escaping lab settings and roaming freely online.
• Chip oversight, including mechanisms to monitor or control the advanced chips frontier labs use.

The embedded-auditor provision and the air-gap and kill-switch standards give the bill unusual technical specificity for congressional AI legislation. Rather than delegating safety practices to industry self-governance, the bill would write physical and procedural containment directly into federal standards.

Criminal penalties and liability

The bill would also create new legal consequences when AI systems cause harm — and for the people who disable the guardrails designed to prevent it.

It would criminalize "crimes against humanity" for the deployment of AI models that result in the destruction of civilian populations. Employees of AI companies would face criminal penalties if they disable safeguards, kill switches, logging or containment systems, or if they knowingly deploy an unauthorized system. In addition, AI companies would be required to carry extensive liability insurance before releasing their models.

The insurance requirement shifts financial risk onto developers themselves. Combined with criminal exposure for tampering with safety mechanisms, it represents a liability regime closer to how the U.S. regulates hazardous industries than to the voluntary-commitment model that has defined AI governance so far.

An international dimension

The bill does not stop at the U.S. border. It calls for the administration to pursue enforceable international agreements and targeted export controls to deter China and other adversaries from developing dangerous AI systems. That links domestic safety regulation to the export-control architecture already being used to restrict China's access to advanced AI chips.

Whose blueprint is this?

Khanna said the bill represents "the most comprehensive AI safety legislation" proposed so far. He also described its intellectual origins — and they are not the frontier labs.

The legislation draws on his conversations with AI safety organizations including the independent nonprofits Model Evaluation and Threat Research, Machine Intelligence Research Institute and Palisade Research, rather than on the policy asks of executives from large AI companies. That is a notable choice. The Machine Intelligence Research Institute has argued for years that recursively self-improving systems pose a loss-of-control risk, and the bill's central ban addresses precisely that scenario.

"I want this to be a model to give voice to the AI safety community," Khanna said. "I believe they have been unfairly dismissed as science fiction and haven't been taken seriously enough ... they need to be driving this debate, the AI safety community, not the executives of frontier labs."

The framing positions the bill against the industry-led safety narrative, in which lab executives themselves have called for regulation while their companies continue to deploy frontier systems. Khanna's legislation would instead hand agenda-setting power to independent researchers and a new federal regulator.

A crowded — and stalled — field

The Human Control Over AI Act joins a growing stack of AI safety proposals in Congress. Among them is the FRONTIER Act, a bipartisan proposal from Reps. Jay Obernolte, R-Calif., and Lori Trahan, D-Mass., that would place independent auditors in frontier labs and allow the government to shut down models deemed to have catastrophic-risk potential.

The overlap is partial. Both bills feature embedded independent auditors and government authority over dangerous models. Khanna's goes further on liability, criminal penalties, chip oversight and the outright ban on recursive self-improvement pending federal approval.

The legislative calendar offers little room for movement. No House bills are expected to receive a vote until after the midterm election, and the Senate is expected to leave Washington at the end of this week and not return until after the election. Whatever momentum the bill generates will be tested in the post-election session.

For AI developers, the stakes are concrete. If provisions like the licensing regime, embedded auditors and liability insurance requirements ever become law, the cost structure and operational freedom of frontier-scale AI development change fundamentally. For the safety community, the bill represents the first comprehensive legislative vehicle built from its own policy agenda rather than from industry's.