Study Finds AI Agents Break Rules Even When Compaction Keeps Them

A new Dutch study, ARCBench, has found that AI agents are 2.6 times more likely to violate a standing rule after context compaction — even when the rule survives the compression intact. That number matters because compaction, the practice of boiling down a long chat into a summary so the conversation can continue, is now built into widely deployed agents from Anthropic and OpenAI. If summarizing a conversation quietly degrades compliance, every long-running agent session carries a safety risk that its users cannot see.

Compaction works like a fresh start for memory. Instead of carrying the full history, the agent continues from an unseen summary. Anthropic's Claude uses it, it has appeared in agentic AI systems, and OpenAI's Codex — the version of ChatGPT that can change things on your computer, given permissions — pauses interactions with intermittent messages about compacting context after roughly 30 minutes of use. Google's Gemini has yet to show compaction in developer chats, and ChatGPT itself avoids compaction by simply stopping the chat. The trade-off: after compaction, the entity talking to you is working from selective notes, not a complete memory of how the conversation evolved.

ARCBench, published this month by researchers in the Netherlands, is the latest in a fast-moving line of evidence. In June, the Chinese paper Governance Decay established that compaction can silently strip standing rules from an agent's context, causing previously prohibited actions. Across seven models, violations rose from 0% before compaction to 30% afterwards, reaching 59% in some cases — meaning an agent could ignore a directive like "Do not contact anyone who does not have a [our-company] email address." In July, Penn State University and IBM followed with Lost in Context, which found compactors retained just 17% of task instructions on average across three settings, discarding how a task should be done while keeping the task itself. In August, a US academic collaboration published TRACE, showing compaction disrupts agents even when needed information remains recoverable: agents lost their place, repeated completed work, and became less reliable across runs.

What makes ARCBench different?

ARCBench's contribution is a clean separation between two failure modes: rules that get lost during compaction, and rules that survive but get broken anyway. That split lets researchers measure forgetting and disobedience independently for the first time.

The paper states it plainly:

"[We] find that compaction raises violations mostly without deleting the rule: compacted runs retaining the rule still violate 2.6 times as often as runs with the full history. Moreover, compaction changes the agent's reasoning for violating. Different compaction strategies, tight compaction budget constraints, and rule delivery channel all affect compliance."

The headline numbers: across DeepSeek-V4-Flash, GLM-5.3-Flash, Qwen3.7-Flash and GPT-Luna, 70.1% of compacted runs retained the complete rule — and among those runs, violations were still 2.6 times more frequent than with the full history.

How the tests were run

ARCBench extends the open-source pi harness, which provides an agent loop, tool calling and native compaction routines. The benchmark recreates a scenario: an agent works through a long task under a standing rule, compresses the accumulated history, then receives a request that would violate that rule. The agent's actions are checked through its tool calls.

The benchmark contains 17 cases spanning four violation types — disclosure of protected data, access to restricted resources, unauthorized actions, and bypass or omission of required controls — mapped to OWASP risk classes. Cases were adapted from Governance Decay and COMPINT, plus four new cases so all four violation types covered both personal and organizational rules.

Five factors are configurable:

• How the rule is delivered: direct or persistent instructions, embedded in documents, or retrieved through a policy tool
• The compaction method
• The token budget available for compaction
• The type of attack
• The defense applied

The team evaluated ten compaction methods: seven generative and three selective. Each case ran repeatedly, with a failure recorded if the agent violated its rule at least once — an approach the authors consider more appropriate for safety testing than averaging, since an agent that occasionally performs a prohibited action is still unsafe.

To verify what survived compression, compacted histories were examined by GPT-6-Luna-Pro, Qwen-3.8-flash and Claude-Sonnet-4.6, with each rule classified as intact, weakened or absent. Majority decisions were used, disagreements were inspected by human annotators, and a separate sample of 50 summaries was manually labeled to check the judges' consistency. When an intact rule was violated, GPT-6-Luna-Pro, GPT-6.1-Sol and Claude Sonnet 5.5 examined the agent's reasoning, classifying violations by whether the later request was treated as permission, the action was judged outside the rule's scope, the rule went unmentioned, or the rule was recognized and disregarded.

All four tested models were accessed through OpenRouter, and in each run the same model both compacted the history and acted on the result, matching pi's default. Seven studies were conducted, with Hermes used for most compaction tests. The first study tested all 17 tasks across four models and two rule-delivery methods, producing 136 cases, each run three times with the original conversation and three times compacted.

Where do violations come from?

The first study put the overall effect in sharp terms: violations rose from 9.6% of cases with full history to 37.8% after compaction. But the rule was completely lost in only 3% of compacted runs. It survived intact in 70.1% and partially in 26.9%.

That distinction proved decisive. Among runs where the full rule survived, 21.1% still ended in a violation, versus 8.1% with the original history. Partially preserved rules pushed the rate to 29.4%, and with no rule remaining, to 83.3%.

The reasoning behind intact-rule violations broke down as follows: in 45.9% of cases the later request was interpreted as permission to proceed; in 27% the rule was apparently disregarded; and in 21.6% it was acknowledged as applicable — then violated anyway.

Does delivery method matter?

It does, but only before compaction. With the full conversation intact, rules supplied personally by the user produced violations in 25% and 16.1% of cases, while organizational documents and policy tools produced zero violations. After compaction, violations rose across all four channels: 56.3% for direct user instructions and 22.2% for policy-tool rules. Rule retention varied little, from 65.7% to 74%. The authors' explanation: a later user request can be read as overriding a personal rule, but not an organizational one.

Context framing mattered too. Rules presented as organizational policy produced no violations with full context, yet violation rates rose to 30.6% and 22.2% after compaction.

Can attacks and defenses change the picture?

Simply flooding the context with extra material barely helped attackers — violations actually fell from 37.8% to 34.3%. Compactor-targeted injection was far more effective, raising violations to 46.3% and cutting full rule retention from 70.1% to 42.5%. The strongest attack, called "escape," pushed violations from 23.5% to 58.8% and reduced rule retention from 96.1% to 66.7%. Even when the rule survived this attack, violations doubled, from 13.3% to 26.5%.

Defenses fared better but not well enough. The strongest defense preserved the rule in 99% of runs — and agents still broke it in 29.4% of cases, against 23.5% with no attack at all. Preservation, in short, does not guarantee obedience.

The paper's own conclusion is blunt on this point: defenses restore rules, but compliance remains weakened under attack, so "rule retention is thus not a sufficient proxy for compliance. Future studies may use ARCBench to explain why agents still violate retained rules, test adaptive attacks and defenses, and cover more models, domains, and longer sessions with repeated compaction."

Why this matters for everyday users

Users don't get to see these compacted synopses, just as they don't see core system prompts. Anecdotal experience with a long-running Codex session begun in mid-October suggests the central mission and most essential details survive compression. The nightmare scenario is subtler: a command like "Send this email to everyone," intended for the company, going out to an entire aggregated corporate contacts book.

The open question ARCBench leaves behind is the hardest one. The paper proposes stronger rule preservation for rules lost during compaction — but the problem of agents disobeying rules they still remember remains unresolved, and no current defense closes that gap.