OpenAI, Google, Microsoft and Anthropic Back a Shared Opt-Out for AI Scraping

OpenAI, Google, Microsoft, and Anthropic have committed to adopting a single technical standard that lets publishers opt out of AI scraping across all four companies at once — instead of negotiating with each crawler separately, as reported earlier by Search Engine Roundtable. Anthropic has already implemented it. Google and Microsoft plan to support it next year. OpenAI says it will roll it out in the "coming months."

The proposal is called a shared "policy token," and it comes from Kevin Kelley, a technical lead for standards at Anthropic. It is being developed through the Internet Engineering Task Force (IETF), the body that standardizes core internet protocols. If adopted widely, it would change a workflow that has become a quiet burden for web publishers: today, a site that wants to keep its content out of AI training sets has to signal its preferences separately to every AI company that sends a crawler its way.

What is the policy token, exactly?

The signal lives inside a website's robots.txt file. That file has long been the internet's do-not-disturb sign, telling web crawlers what they can and can't scrape — whether the purpose is search indexing, archiving, or, more recently, AI training. The policy token extends that mechanism. When a publisher sets its AI training preference once, the token carries that instruction in a form all participating AI companies have agreed to recognize.

The four companies — OpenAI, Google, Microsoft, and Anthropic — say they will restrict their AI crawlers from accessing a site when they encounter the shared token. That is the core commitment. One file edit, one preference, applied across the industry's dominant AI players.

Microsoft, however, is adding a wrinkle. The company will offer more granular controls than its peers. Publishers working with Microsoft will be able to opt out of AI training or, separately, out of AI grounding — the process of pointing an AI model to a verified source of information to reduce hallucinations. A site could, in principle, allow one and refuse the other. Microsoft also notes that websites will still appear in Bing's search results regardless of their preferences. That carve-out matters for publishers who fear that blocking AI systems means disappearing from search entirely.

Why does this matter for publishers?

The announcement lands amid a persistent problem for anyone who publishes on the open web. AI crawlers have been scraping sites at scale and feeding their content into AI models, often without payment or permission. Publishers have been forced to improvise defenses.

Two efforts stand out as context. The Really Simple Licensing (RSL) standard gives publishers the ability to charge crawlers that visit their sites — turning unwanted traffic into a billing event rather than a silent extraction. And Cloudflare, one of the largest internet infrastructure providers, now blocks AI crawlers by default, flipping the default posture of the web from open to guarded for a vast share of sites.

The policy token fits into this same shift. It does not charge anyone anything, and it does not block anyone by default. What it does is simplify the paperwork. Under the current fragmented system, a publisher must track down the crawler-signaling conventions of each AI company and configure them one by one — a maintenance burden that grows with every new model provider entering the field.

With the token, that fragmentation collapses into one instruction. And because the proposal runs through the IETF rather than through any single company's documentation page, it carries the weight of an open internet standard — the same process that produced many of the protocols the web runs on.

Who has actually shipped it?

The timelines diverge sharply, and the divergence is part of the story.

• Anthropic: already adopted the signal. It is the only one of the four that has implemented it, and notably, the proposal's author — Kevin Kelley — works at Anthropic as a technical lead for standards.
• OpenAI: says it will implement the token in the "coming months," per the company's statement.
• Google: plans to support it next year.
• Microsoft: plans to support it next year, with its granular training-versus-grounding controls attached.

That staggered rollout means publishers who deploy the token now get immediate protection only from Anthropic's crawlers. Full coverage across the four companies arrives over a longer horizon.

Does this resolve the scraping fight?

Partially, and only for these four players. A shared opt-out standard answers the mechanical question of how a publisher says no. It does not answer the harder questions that have defined the past two years of publisher–AI relations: whether scraping for training should require permission in the first place, whether publishers should be compensated for content that ends up in model outputs, and what happens with AI companies that never sign on to the standard at all.

It also does not address the enforcement gap. Robots.txt and its successors are honor-system technologies. A crawler that ignores the file faces no technical barrier, only a policy violation — which is why complementary approaches like Cloudflare's default blocking and RSL's pay-to-crawl model exist in the first place. Publishers who deploy the policy token are trusting that OpenAI, Google, Microsoft, and Anthropic honor it, and that list of four, while dominant, is not exhaustive.

Still, the coordination itself is the news. Four companies that compete fiercely for model supremacy — and that have faced mounting criticism and litigation risk over training data practices — have agreed on a common machine-readable way to be told "no." For a standards process like the IETF, getting this quartet aligned before formal adoption is a meaningful signal of intent.

What happens next?

Watch the IETF process. The policy token is a proposal moving through standardization, which means it can be revised, refined, or delayed as other stakeholders — including publishers, infrastructure providers, and rival AI companies — weigh in. The pace of OpenAI's "coming months" implementation, and whether Google and Microsoft hit their next-year targets, will determine whether the token becomes a real industry norm or a well-backed pledge.

For publishers, the practical move is straightforward once support goes live: set the preference once in robots.txt, and it propagates. For everyone else, the story is a milestone in the tug-of-war over training data — the first time the biggest AI companies have jointly agreed to make "no" easier to say.