OpenAI's Codex agents now fix ChatGPT performance bugs end to end
OpenAI says its Codex-powered 'Performance Factory' investigated 1,700+ ChatGPT performance regressions in three months, root causing 600 — with adversarial agent reviewers and a 70% diagnosis accuracy rate.

Why it matters
- The Regression Workflow tracked more than 1,700 investigations and root caused over 600 in just over three months — 17 per day.
- A manual review of 100 agent root cause suggestions graded 70% accurate; 30% needed refinement or didn't establish causality.
- The system uses adversarial Codex review sessions that try to disprove each diagnosis and each proposed fix.
- Engineers define workflows in Markdown; agents run in the cloud under scoped service identities with access to repos and telemetry.
- OpenAI is developing proactive regression detection and exploring agent-shipped fixes without a human in the loop.
OpenAI's autonomous agents have investigated more than 1,700 performance regressions in just over three months and root caused over 600 of them — a pace of 17 automated investigations every day. That is the headline number from OpenAI's engineering team, which has published a detailed account of the Performance Factory, a cloud-based system that uses its Codex coding agents to find, diagnose, fix, and verify ChatGPT slowdowns with limited human involvement.
The disclosure matters beyond OpenAI's infrastructure. It is one of the most concrete public descriptions of agentic AI running a core engineering function in production at scale — not a demo, not a pilot, but the machinery that keeps a product serving hundreds of millions of users responsive. As agentic development accelerates across the industry, OpenAI is effectively arguing that manual performance engineering can't keep up, and that the fix is software that debugs itself.
What is the Performance Factory?
The Performance Factory is OpenAI's automated system for performance engineering, running entirely in the cloud. Engineers define workflows and write instructions, in Markdown, for Codex agents to follow. The system has four components:
- a task tracker with configurable workflows;
- a dispatch loop that assigns ready work to agents;
- an internal API that runs Codex turns in the cloud;
- event subscriptions that wake agents when something changes, such as a GitHub review comment or a completed deployment.
Each piece of work becomes a task with its own state and findings, so work passes between agents and engineers and can resume after a code review or a deployment. Agents run under a scoped service identity, with access to OpenAI's code repositories and the same observability tools engineers use — metrics, logs, traces, CPU profiles, and memory snapshots.
How does the Regression Workflow actually work?
The first and so far only fully running workflow handles performance regressions. When a configured performance alert fires, the workflow opens a task. An agent investigates the slowdown, records its findings, and proposes a fix for an engineer to review. After deployment, another agent picks up the same task to check whether performance has returned to baseline. From investigation to fix to verification, OpenAI says the entire regression-fixing process is now automated.
The investigation itself follows a forensic playbook. The agent first works out which metrics got worse, when they started to change, and which service requests were affected. Since a regression may have started hours before the alert fired, the agent looks back through the data to find the true starting point. It compares groups of requests to determine whether similar requests actually slowed down, or whether the system was simply serving more already-slow requests — findings that point to different causes.
The agent then examines deployments and feature flag changes, reads the relevant code, and uses logs and traces to test whether a change could explain what happened.
Why did OpenAI add an adversarial reviewer agent?
Early on, agents sometimes stopped at the first plausible explanation, and some of those diagnoses turned out to be wrong. OpenAI's answer was an adversarial review: a separate Codex session that tries to disprove the proposed diagnosis by rechecking assumptions and testing counterfactuals. If a rollout caused the slowdown, did similar requests that weren't exposed stay healthy? If the suspected change was rolled back, did the metric recover?
The company shared a concrete case. An agent traced failing ChatGPT conversations to errors in image downloads and marked the cause as found. The Codex reviewer noticed that the internal service responsible for locating files had already recovered while image requests were still failing. The diagnosis didn't explain the continuing failures, so the reviewer reopened the investigation.
The accuracy numbers are honest rather than flattering. In a manual review of 100 root cause suggestions, OpenAI graded 70% as accurate, with 30% needing refinement or failing to fully establish causality. The workflow layers checks and balances — separate turns investigate, challenge, and fix — to push those numbers up.
What happens after the diagnosis?
Once a diagnosis passes adversarial review, the workflow asks Codex to implement a fix and open a pull request. An engineer reviews it and decides whether to merge. Each task carries an auto-generated report — regression summary, impact, root cause, proposed fix — tuned for scanning, with metrics time series, CPU flame graphs, and traces. If the engineer leaves PR comments, the workflow wakes an agent to address them.
After deployment, an agent verifies recovery. Full recovery closes the task. Partial recovery resolves the original regression and opens a new one for the remaining slowdown, which may have a different cause. No improvement sends the whole thing back for investigation.
OpenAI is candid that fixes are harder than diagnoses. Agents have proposed changes more complicated than necessary or ones that introduced new issues. A second adversarial review now checks each proposed fix. Some fixes also need architectural judgment — adding a cache might improve performance but introduce stale data — and OpenAI says those calls still belong to an engineer who knows the service.
How do you trust an agent's work at scale?
The team's operational lessons read like a manual for running agents in production. Raw Codex output can be a wall of text, so summaries are engineered to let engineers spend attention on decisions, not on reconstructing the agent's work. Agents can investigate more regressions than a team has time to read, so the workflow links related incidents, groups them by the metrics teams care about, and ranks estimated impact.
Every skill update or model upgrade can make agents better at some cases and worse at others, so OpenAI backtests against past production incidents. Each test starts with a fresh Codex investigator and the original signal, stripped of spoiler details — no saved memories, no incident writeups, no previous run results. The team scores timing, magnitude, affected requests, and how the agent ranks causes and expresses uncertainty. An agent can nail the timing and still blame the wrong deployment, so those are scored separately.
Even the agents get monitored. A broken profiler or a missing permission can silently block an investigation, so OpenAI runs a daily agent that scans recent runs for tool failures and access errors — and then uses the Performance Factory itself to fix the problems it finds.
How did the team get here?
OpenAI describes a six-stage evolution. Before Codex, engineers manually examined metrics, logs, and code, accumulating more than 30 performance tools over time. Codex then became an investigation partner that could search large codebases and run profiling on its own. The team captured its investigation process in a shared skill called perf-forensics, tested with evals built from past regressions. Engineers began to feel like copy-paste proxies between alerts and Codex, so investigations moved to the cloud, firing automatically from alerts. The Performance Factory completed the arc: end-to-end workflows with state, handoffs, and verification.
What comes next?
Today the Regression Workflow starts when an alert fires. OpenAI is building proactive detection — agents mining fine-grained CPU and memory profiles, heap dumps, and request traces across releases to catch regressions its alerts miss, or spot optimizations nobody was looking for. A service rightsizing skill already helps agents flag autoscaling settings that keep more replicas running than a service needs.
"Next time you're using ChatGPT and notice a slowdown, there's a good chance the Performance Factory is already figuring out why," the team writes. Today the bottleneck is waiting for an engineer to review and merge the fix. OpenAI says it is already exploring whether agents can do that themselves — and as product development accelerates, the company's position is explicit: performance systems must become increasingly autonomous and self-improving to keep up.
Source: OpenAI News
More from Elena Vasquez
Show full bio
Market editor covering technology and product news in artificial intelligence and machine learning at AI In Context.
795 articles
Related articles
- OpenAI Ships AgentKit, Expanded Evals, and Reinforcement Fine-Tuning for Agents
- OpenAI Models Are Coming to Amazon Bedrock via a Stateful Agent Runtime
- OpenAI says Codex runs 'daily' across its engineering teams
- Cloudflare Brings OpenAI's GPT-5.4 to Agent Cloud for Enterprises
- OpenAI ships a model-native harness and native sandboxes for its Agents SDK