OpenAI's textGrain Watermark Falters on Short and Edited Text

Replacing just 10 percent of a watermarked passage's words with synonyms cuts OpenAI's detection rate from 92 percent to 66 percent, according to the company's own tests of textGrain, the watermarking system released in October to comply with the EU AI Act. Replace 25 percent of the words, and detection collapses to 17 percent.

The numbers matter because the EU AI Act's transparency obligations, which took effect on 2 August under Article 50, require providers of systems that generate synthetic audio, images, video, or text to mark outputs in a machine-readable format so they can be identified as artificially generated or manipulated. The law demands technical solutions that are "effective, interoperable, robust and reliable as far as this is technically feasible" — but it sets no measurable benchmark for how well a watermark must perform. That gap leaves OpenAI exposed: if textGrain is eventually found non-compliant, the company could face a fine of up to 7 percent of its annual income, says Thomas Souverain, a postdoctoral fellow on AI Ethics at the Institute of Advanced Education in Paris.

How does textGrain work?

Language models calculate the probabilities of different possible next tokens — a token is the next word or word fragment the model generates — based on the text already written, then select one and repeat the process until the response is complete. After the prompt "today the weather is…", the model might weigh "warm" or "cool" as candidate next words, says Xiang Li, co-author of the textGrain technical report and a postdoctoral researcher at the University of Pennsylvania.

A watermark uses a secret key to generate pseudorandom values — numbers produced by a mathematical process — that influence which token the model suggests. This creates a statistical pattern in the text that a detector holding the same secret key can search for, Li says.

What distinguishes textGrain from other watermarks is its use of entropy, a measure of how unpredictable a model's word choices are, as a budgetable resource. Developers specify how much entropy, or unpredictability, they want to allow, and textGrain adjusts to that target. Li puts it in concrete terms: "We can think of entropy as a resource, a kind of money. […] In our approach, we can accurately determine how much of that resource we want to consume."

This design gives developers more control over the trade-off between embedding a detectable signal and preserving variation in the model's output, Li says.

How reliable is detection in practice?

The watermark is invisible to readers, but its results carry caveats. A positive result does not show how much a person contributed to a passage. A negative result does not prove a text was human-written.

OpenAI's false-positive rate is 1 percent, meaning the detector is designed to wrongly flag about one in 100 unwatermarked passages. In tests on psychology questions from the ELI5 database, which gathers questions and answers from the /r/explainlikeimfive subreddit, the model detected watermarks in about 80 percent of 200-token text — approximately 150 words — compared with 95 percent of 400-token text, approximately 300 words.

Li calls the short-text problem a "basic limitation" of text watermarking technologies and says it would be complicated to solve. Short passages may not contain enough watermark evidence for reliable detection, even with additional computation, he adds. Longer passages generally provide more.

Editing weakens the signal further. Because the watermark depends on preceding tokens, changing one word can disrupt the signal at several positions, Li says. The synonym-replacement results — 92 percent detection dropping to 66 percent at 10 percent replacement, then to 17 percent at 25 percent — come from OpenAI's evaluation of 400-token passages.

OpenAI itself says watermarking alone is not enough to determine whether content is AI-generated. The company positions textGrain as a complement to other provenance tools, such as content credentials embedded in metadata and the SynthID watermarks developed by Google DeepMind for images and audio. OpenAI did not respond to a request for comment.

What does the EU AI Act actually require?

Article 50's transparency obligations went into effect on 2 August. Providers must mark synthetic outputs in a machine-readable, detectable format. The EU's Code of Practice offers a voluntary route for providers to demonstrate compliance — and, notably, it does not require watermarks on outputs shorter than 200 tokens, a carve-out that loosely matches textGrain's weak spot at around 150 words.

Souverain says the legislation is open to interpretation. He considers the early methodology behind textGrain reliable and says it could work better than competitors at identifying AI-generated text. OpenAI claims textGrain matched or outperformed Google's SynthID Text in its own tests, but those results have not been independently verified.

Until independent researchers can test the system, and while the AI Act's criteria still lack measurable thresholds, Souverain says it is too early to say whether textGrain meets legal requirements. "We only have a technical study … there hasn't been a peer-reviewed academic article, and we don't [yet] have access to a precise protocol [for textGrain]," he says.

Who is testing it, and what happens next?

OpenAI is working with a limited group of external researchers. That group includes John Thickstun, an assistant professor of computer science at Cornell University, and researchers at the Kempelen Institute of Intelligent Technologies (KInIT) in Bratislava, Slovakia.

Thickstun and Dominik Macko, a senior researcher at KInIT, say they had only just gained access and had not yet begun evaluating the system. Macko says his team plans to test textGrain's "robustness" across different subject areas, languages, and text lengths — the exact dimensions where editing and brevity already erode detection rates.

Souverain calls on OpenAI to make textGrain open-source so researchers outside the company's circle can "identify all the faults, paraphrases and possible translation faults" of the model by attempting to break it. "I see the corporate interest, if there is a big flaw detected, a big weakness, [that] it is detected in the background," he said. "But I would like to see an even more open phase, such as today with SynthID, [where] we have access to the methods."

The stakes extend beyond one company. With no clear legal benchmark for watermark performance, and detection degrading under basic editing, the EU's transparency regime currently rests on tools whose limits are documented by their own developers. The independent evaluations at KInIT and Cornell, once under way, will provide the first external evidence of whether textGrain can survive contact with real-world use — and with the AI Act's still-unspecified standards.