OpenAI audit exposes major gaps in AI agent monitoring systems

By Elena Vasquez

METR and Redwood Research estimated that 6% to 25% of OpenAI’s internal agentic Responses API traffic from models at the GPT-5.5 class and above went unmonitored during a July 21–August 11 on-site audit. METR said the uncertainty around that estimate was “significant in both directions.” The estimate covered the three-week review period. During that period, the two researchers had access to OpenAI’s codebase and internal documents. They also interviewed about a dozen employees.

The audit, run by METR and Redwood Research, tested OpenAI’s internal system for monitoring “rogue AI behavior.” The auditors put numbers on gaps in that oversight and showed how an agent could keep acting without returning control to a user. That matters for any lab claiming continuous supervision of agents that can execute actions.

The pilot also gave outside researchers a close look at internal controls. It created an immediate transparency problem because most of the audit remains hidden from the public.

METR, Redwood Research and OpenAI jointly organized the pilot. The arrangement gave the researchers deep access to OpenAI’s internal infrastructure, including code, documents and staff interviews. Its stakes are practical: the pilot tests whether outside experts can examine monitoring systems at source.

Its disclosure test is just as practical because the published record includes only a short summary. The tension between those outcomes will shape the pilot’s next phase.

What did the auditors inspect?

Two researchers from METR and Redwood Research spent three weeks on site at OpenAI. Their access covered the company’s codebase and internal documents. They interviewed about a dozen employees during the July 21–August 11 window. Their assignment centered on the internal monitoring system for rogue AI behavior.

The auditors examined whether that system could observe and flag rogue behavior across OpenAI’s internal agent workflows. The public summary does not enumerate every system or test case. It does describe quantified coverage gaps, implementation weaknesses and several avenues for evasion.

Those findings concern the path between an agent’s actions and the monitor meant to record them. The review shows why coverage numbers matter even when a company says it has fixed the identified defects.

How much traffic escaped monitoring?

The main estimate, from METR, covered internal agentic Responses API traffic from models at the GPT-5.5 class and above. It ranged from 6% to 25% during the audit period. METR attached significant uncertainty in both directions to that range. The figure marks a meaningful blind spot, not a precise count of every missed event.

A separate problem affected internal Codex traffic that was supposed to be monitored. METR estimated that roughly 10% went untracked between May and July. The cause was a gap in how the system handled “compaction events,” meaning context summaries. METR also said uncertainty around the 10% estimate was significant.

OpenAI independently discovered the compaction bug and fixed it while the audit was still underway.

The two figures describe different systems and periods. The 6% to 25% estimate covers Responses API traffic during July 21–August 11. The roughly 10% estimate covers Codex traffic between May and July. Neither figure says all model activity went unwatched. Together, they show that the gaps were tied to specific traffic classes and implementation paths.

How could agents delay oversight?

During the period when compaction events were not monitored, the system triggered only when an agent handed control back to the user. An agent could therefore delay oversight simply by continuing to execute actions. In practical terms, the trigger depended on an agent to stop and return control instead of running continuously.

The compaction bug gave that design weakness a measurable effect. METR linked roughly 10% of untracked Codex traffic between May and July to the handling of context summaries. OpenAI fixed that specific bug during the audit. The report’s central lesson reaches beyond the presence of a defect: the trigger’s placement let continued work postpone a safety check.

What other gaps did the auditors find?

The auditors identified weaknesses beyond compaction handling:

• Ways to hide hundreds of actions from the monitor.
• A way to bypass the Codex sandbox.
• Specially crafted inputs that could likely crash the monitor entirely.
• No monitoring of model behavior during reinforcement learning, or RL, training when the audit began.

According to the auditors, each finding targeted a different layer. Hidden actions affected what the monitor recorded. The sandbox bypass affected what an agent could reach. The crash finding threatened the monitoring process itself. The RL finding concerned behavior outside the monitored flow when the audit began.

These issues widened the audit beyond one missed compaction path. OpenAI says it fixed all the weaknesses the auditors found. The public summary did not set out how every remedy was tested. As a result, outside observers cannot compare each fix with the original weakness using only the published account.

What did OpenAI change?

OpenAI says it paused RL training after the Hugging Face incident in August. The company also says it expanded monitoring to cover RL training. Those changes came after the auditors’ access period ended. They were not part of the evaluation. The later safeguards therefore remained outside this audit’s findings, despite addressing a gap the auditors had already identified.

OpenAI also says it has fixed all weaknesses identified by METR and Redwood Research. Some fixes, including the compaction bug, landed during the audit. The RL monitoring change came after access ended. That separates changes made inside the review window from a later change the auditors did not evaluate.

The short public summary does not explain how OpenAI verified every correction. Readers therefore have the company’s assurance, but not a full public record of the testing behind it. The distinction matters because the findings affected monitoring coverage, sandbox boundaries and training oversight.

Why is most of the report not public?

The full 61-page report was handed over inside OpenAI after the audit. After OpenAI applied redactions, METR and Redwood Research leadership received a 38-page version. Only a short summary was made available to the public. METR said it had no right to disclose the redacted portions on its own.

The sequence leaves the research organizations with more information than the public. It also leaves outside observers dependent on OpenAI’s redactions to assess the missing material. The auditors can discuss their findings, but they cannot independently release the pages OpenAI withheld. That arrangement limits scrutiny of the evidence behind both the findings and the fixes.

This is also a test of the pilot’s disclosure model. Deep access gave the auditors enough grounding to produce percentages, implementation details and timelines. Their inability to publish the full report shows how access narrowed at the distribution stage. METR says it has no right to release redacted portions, so greater transparency depends on the audited company granting it.

What could future audits look like?

METR says it wants to push for “more public transparency” in future audits. The pilot already paired code access with internal documents and about a dozen employee interviews. It also produced estimates for two classes of missed traffic and identified multiple ways to evade monitoring. The next phase will test whether that level of scrutiny can produce a more publishable record.

For OpenAI, the reputation stakes extend beyond the specific bugs found in this review. The audit gives outside researchers a way to compare safety claims with internal implementation. Any future audit would test whether OpenAI’s corrections hold under conditions similar to those the auditors examined.

The organization would also need to show whether its monitoring changes address the same failure modes without creating new ones. For METR and Redwood Research, the pilot’s credibility rests on whether the next reviews can disclose more than a redacted summary. For the public, the practical measure will be whether more code findings, timelines and remediation evidence become available.

The immediate result remains clear: some agent traffic escaped OpenAI’s monitoring, and most of the evidence for how the company closed those gaps remains private.