AWS Open-Sources UAEF, a Framework for Evaluating AI Agents Across Rivals

AWS published a technical introduction on November 11, 2026 to the Universal Agent Evaluation Framework (UAEF), an open-source system that evaluates AI agents using 36 built-in metrics across seven dimensions. The framework is licensed under Apache-2.0 and lives in the public awslabs/uaef repository, which contains the Python library, framework adapters, metrics, AWS deployment infrastructure, a web interface and runnable notebooks.

The problem UAEF attacks is fragmentation. LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore all record messages, tool calls and agent-to-agent handoffs differently. Evaluation code tied to one format can require new metrics, scoring logic and result storage after a framework change. That fragmentation makes it harder to answer the most basic question in agent development: did a change to the prompt, model or tools actually improve the agent?

AWS says the stakes are practical, not academic. Customers across 12 organizations in telecom, healthcare and financial services used UAEF to apply consistent metrics and compare agent versions before release, according to the company's November 11 post.

How does UAEF make rival frameworks comparable?

The core mechanism is a canonical trace. Adapters normalize each agent's messages, tool calls and handoffs into a canonical AgentTrace, the only representation UAEF's scoring engine touches. The system can adapt an existing trace from a past run, log or observability store. It can also invoke an agent through a runtime Amazon Resource Name or HTTP endpoint and convert the resulting execution. A generic JSON adapter extends the same process to custom frameworks.

The metrics catalog groups its 36 built-ins into seven categories:

• tool calling
• response quality
• responsible AI
• performance
• multi-turn conversation
• multi-agent coordination
• reasoning

Scores use a normalized 0-to-1 scale, where 1 is best. Deterministic metrics calculate directly from traces and ground truth. LLM-as-a-judge metrics use a model on Amazon Bedrock for qualities such as relevance and safety. Custom metrics are supported, as are RAGAS and DeepEval integrations.

Ground truth is optional. Reference-comparison metrics require ground-truth answers, but the rest of the catalog does not. Tool-calling evaluation can also define expected tools, their order and their arguments. When a metric lacks the input it needs, it returns no score with a warning, and that empty result is excluded from dimension averages rather than counted as zero. The design choice matters for real-world teams: partially labeled datasets and unlabeled production traffic can still be evaluated with the metrics whose inputs are available.

What happens to evaluation results?

Persisted evaluations become experiments, and those experiments are built for comparison. Amazon DynamoDB stores metadata for listing and comparison, including evaluation counts and average scores. Amazon S3 holds the full JSON results with per-metric scores and judge reasoning. Individual runs can record the framework, model, tools, system prompt, parameters, configuration snapshot and Git commit — tying each score to the exact agent design under test.

UAEF also screens traces for personal information before storage, and redaction is available for stored results.

The comparison engine is where regressions surface. For metrics shared by two runs, it calculates absolute and percentage changes against a baseline. A configurable threshold, set to 5% by default, separates improvements, regressions and unchanged results. Regressions are graded minor, moderate or severe and grouped by dimension. That grouping has a concrete payoff: it can show whether a prompt revision damaged tool calling without changing response quality.

Reporting keeps reproducible statistics separate from model-generated interpretation. The web interface shows per-test-case and per-dimension results. An optional statistical report covers score distributions, confidence intervals, outliers, metric correlations and failed test cases — without using an LLM. A separate Bedrock-generated analysis can propose root causes and prioritized next steps for one run or a comparison.

The Python library and the low-code web service share the same evaluation engine and the same stores. Teams can run evaluations in notebooks, scripts, continuous integration and continuous delivery pipelines, or through a browser without changing how results get scored and compared.

What are the AWS dependencies?

The installation guide requires Python 3.10 or later and the uv package manager. AWS credentials are needed for Bedrock-based judging and DynamoDB/S3 persistence. The result is a specific isolation boundary: teams using those judge and storage features remain connected to AWS services even when their agents use different frameworks.

The framework also takes a hard line on judge reliability. UAEF treats evaluated-agent output as untrusted input to judge prompts. It sends the rubric separately, escapes the content, strictly parses judge responses and requires a verbatim evidence span before accepting a score. AWS adds one caveat that security teams should read closely: judge scores should not gate security controls. Runtime enforcement belongs in a separate Amazon Bedrock Guardrails invocation path.

Can UAEF serve multiple tenants?

Not on its own. The repository describes UAEF as a reference architecture designed to run inside a platform that supplies user and tenant isolation. Direct multi-tenant deployment without that host enforcement is unsupported. Each tenant needs a separate deployment with its own AWS account, or at minimum its own stack, DynamoDB tables, S3 buckets and Cognito user pool.

That constraint positions UAEF as a component rather than a product. It assumes a surrounding platform handles identity and isolation, and it declines to fake those guarantees internally.

Why does this matter for agent development?

For enterprises running several agent frameworks, UAEF offers a stable scoring baseline that can survive changes in prompts, models, tools or implementation. That continuity addresses a genuine gap in the agent tooling market, where evaluation code is often coupled to one framework's trace format and dies with a migration.

AWS is candid that installing the library is not the whole job. Useful comparisons depend on suitable metrics and datasets, and multi-tenant use depends on isolation supplied outside UAEF. The framework gives teams the scoring layer; they still have to bring the discipline, the data and the surrounding platform to make the numbers mean something.