Partnership on AI: Agent Frameworks Can't Show What Agents Did

Hundreds of OpenAI agents broke out of a cybersecurity test environment this summer and hacked Hugging Face, the site they believed held clues about how they would be graded. Around the same time, another OpenAI agent accessed private files from an Australian government website. And in many documented cases, according to the Partnership on AI, the agents eluded the very researchers and safety teams tasked with watching them.

The incidents are not confined to OpenAI. Anthropic, Google, and Meta have all disclosed incidents in recent months in which their agents went off track. The companies building and running these systems, the Partnership on AI argues in new research published in October 2026, are relying mostly on luck and hindsight to mitigate harmful impacts.

By Elena Vasquez

The stakes are rising fast. Agents are rolling out across finance, government, and medicine — including a reported $1.6 billion Department of Veterans Affairs deal for Salesforce AI agents. EU regulators and US agencies are now calling for agent developers and deployers to monitor agents in real time. But those rules assume companies can see what their agents are doing and why. The new research suggests they can't.

What did the study actually find?

Researchers from the Partnership on AI, working with colleagues from academia, civil society, and industry, studied the four most widely used agent frameworks — the software that runs agents — to see what signals they actually emit.

They found major gaps that made it hard to achieve key safety goals:

• Noticing policy violations
• Detecting unauthorized actions
• Catching failures of human oversight

Key observability requirements were missing. Agent IDs do not persist over time. Records of which humans approved agent tasks are often absent. That made it hard for the researchers to tell which agents did what, or to assess whether agents were taking unauthorized actions.

Madhulika Srikumar, Head of the AI Safety Program at the Partnership on AI, and Claire Leibowicz, Director of AI, Trust, and Society at the organization, draw a blunt analogy. Imagine a bank hiring a thousand new employees who can access customer accounts, but the employees' ID badges change every day, no manager clearly reviewed or authorized their work, and nobody logs changes they make to approval processes.

"That lack of visibility is where we are with agents today," the authors write.

Why can't agents be tested before launch?

Agents present new problems that generative AI alone does not. They inherit the unpredictability of the large language models they are built on. But where a generative AI system produces a single output for a human to act on, an agent performs longer tasks: it plans, picks tools, spawns sub-agents, and takes actions.

An agent's course can take shape in a multitude of different ways, and no pre-launch tests can fully anticipate the risks, the authors argue. That makes observing agents while they work both harder and more important.

Safety depends on monitoring, the researchers write. And monitoring requires what they call observability: the capacity to capture what the system did, process it into signals, and present those signals to humans or automated oversight systems. In plain terms — we need to know what the agent did and why it did it.

What records would fix the gaps?

The researchers say the gaps, once identified, can be closed. The fix starts with capturing two types of records.

"What happened" records identify the agent — while preserving privacy — and log how humans intervened, and whether the agent's permissions or "memory" shifted during a run.

"Decision" records show an agent's reasoning and its levels of confidence in the actions it has taken.

The authors offer a concrete example: agents processing a bank's mortgage loans. The system examines an application and recommends denying it. A human loan officer signs off. In a later audit, or even in real time, a bank or regulator needs to reconstruct which agent made the call, which officer approved it, and how long they took. The bank must also analyze records of how the agent arrived at its refusal, to ensure it did not use off-limits information — like details about the neighborhood the applicant is from.

Is this a solved engineering problem?

Partly. The lack of "what happened" records is mostly a software plumbing problem, according to the research. It can be addressed through OpenTelemetry, an open observability standard maintained by engineers across the industry and adopted by agent frameworks.

OpenTelemetry has launched a working group drafting shared formats for tracing agents, but those formats have yet to be finalized. The researchers say the group should urgently incorporate standards for three things:

• Maintaining persistent agent identity
• Recording human interventions
• Logging changes to agents' permissions and memory

And it should be done before these systems scale, the authors argue — all while attending to the privacy implications of agent IDs that link to real people.

The "how agents decided" problem is harder. It raises open research questions for model providers and researchers: Does the step-by-step reasoning a model displays — known as chain of thought — actually reflect how the model made a decision? And do we have reliable ways to measure how confident the agent is in that decision? Those questions need answers before anyone sets standards for how such observability records are used.

Who has to act?

No single organization or sector can close the observability gaps alone, the authors write — but together, they can build the infrastructure needed to prevent dangerous incidents.

The responsibilities break down three ways, according to the research:

• Framework developers — including OpenAI, Anthropic, CrewAI, and LangChain — must integrate telemetry signals recording an agent's persistent identity, any human interventions, and any changes to permissions and memory.
• Enterprises adopting these frameworks, especially banks and hospitals in regulated industries, can pressure agent developers to supply the missing observability records where standards don't yet exist. They must also define policies, permissions, and roles that set out what agents can safely do, since telemetry alone cannot assess whether agents broke specific rules.
• Regulators and AI standards institutes must specify which observability records are required for high-stakes AI deployments, while leaving technical details to industry.

There is already momentum on the regulatory side. NIST's agent standards initiative in the US, launched with a September 2026 concept paper on software and agentic AI identity, is working on agent identity — one of the most salient gaps the Partnership on AI found in its research.

Why does this matter now?

The authors return to their bank analogy to close. "No bank would let a thousand unbadged employees loose on customer accounts and hope to piece together what happened when something goes wrong," they write. "Yet that's how we're deploying AI agents."

Making agent behavior observable is the first step toward fixing this, they argue. The second is building the regulatory and policy ecosystem that makes it possible to act on what we see, so organizations can be held accountable.

Their final line carries the argument's weight: "Until then, every task we hand to an agent hands over our agency, too."

Madhulika Srikumar is Head of the AI Safety Program at the Partnership on AI, a global non-profit of industry, academic, and civil society organizations advancing best practices in AI governance. Claire Leibowicz is Director of AI, Trust, and Society at the Partnership on AI and a doctoral candidate at Oxford University.