Pope's AI Adviser Calls Big Labs' Rule-Making a 'Cartel'

Father Paolo Benanti, the priest who advises the Catholic Church on artificial intelligence, says the world's largest AI labs operate like a "cartel" — using warnings about superintelligence to shut everyone else out of decisions about how the technology should be governed.

"Ethics cannot be something that belongs to the board of trustees of a company. It's something that has to be balanced, mitigated in debate in the public square," Benanti told WIRED. "We need democratic regulation."

His intervention lands at a moment of unusual turbulence for the AI industry. Warnings from researchers inside major labs about rapid development have fueled public alarm that AI could one day extinguish humanity. In an essay published on the heels of those safety warnings, Anthropic CEO Dario Amodei called for mandatory third-party evaluations of new AI models before release and a crackdown on distillation — the practice of training rival models on the outputs of proprietary systems. Others have argued for a total ban on recursive self-improvement, where AI helps build future versions of itself.

Benanti takes a different position. He is less interested in restricting how AI is developed — methods-focused rules, he fears, could quash innovation — and more focused on standards that guarantee models cannot slip from human control. And whatever those standards look like, he insists, the labs themselves should not write them.

"The narrative of the frontier labs is self-interested, to gain or maintain a position in the market," Benanti said. When a small group of companies dictates the rules under which an industry operates, he argues, "this has a really clear name: a cartel."

Benanti is skeptical of grand pronouncements about model capabilities, citing the overlapping economic, political, and personal incentives at play. OpenAI and Anthropic have both voiced support for slowing development while preparing to go public, with many billions of dollars at stake for whoever captures the market. The rivalry between the US and China — where the most advanced models are being built — further complicates the picture. Anthropic declined to comment. OpenAI and Google DeepMind did not respond to requests for comment.

He also objects to the quasi-religious reverence with which parts of the industry discuss superintelligence, framing it as sophisticated beyond human understanding and practically omnipotent. OpenAI CEO Sam Altman once said he aspires to build a "magic intelligence in the sky." That framing, Benanti argues, distorts the discourse.

"Are we talking about something, or some-one? This is the real difference. It's not a matter of faith, it's a matter of mathematics," he said. "We are not building the next divinity. We are simply deluding ourselves that there is something more than statistical computation [going on]."

Benanti does not formally represent the Holy See, the Catholic Church's governing body. He counsels the church as a scholar of philosophy, theology, and technology. He told WIRED he uses AI to build tools and materials for his university students, but declined to say whether Pope Leo XIV engages with the technology, citing what his staff described as an "oath of secrecy."

The Vatican's engagement is nonetheless unmistakable. In May, Pope Leo published his first encyclical — a letter applying religious teaching to contemporary problems — and chose artificial intelligence as its subject. The Pope warned against the technocratic logic running through the AI industry, which presents "efficiency as the ultimate measure of value" and casts human beings as "a project to be optimized." He called for international collaboration between rival superpowers and AI companies, urging them to set aside the "desire to secure geopolitical or commercial dominance," and warned of "the danger of humanity becoming a victim of its own achievements."

Benanti argues that the flap over superintelligence and existential risk risks distracting from the encyclical's ambitions, by encouraging the public to forfeit its say over the technology's direction to a small group of mathematicians, computer scientists, and executives.

"The first thing is, we need to have brakes, and a wheel," he said. "That means a significant position of control for human beings over the machine."

With the EU's regulatory regime still settling and US policy contested, the question of who designs AI's guardrails — the companies building the systems or democratic institutions — remains the industry's most consequential unresolved fight.