LangSmith, Langfuse, Opik and Braintrust All Built AI Gateways in 2026

Four LLM observability vendors — LangChain, Langfuse, Comet and Braintrust — all shipped AI gateways between May and November 2026, moving from recording model calls to blocking them before they reach a provider. The shift turns tracing companies into runtime infrastructure, and it raises the question of who inside a company owns the path every model call travels.

What changed in six months?

LangChain announced the LangSmith LLM Gateway in private beta in May 2026 and opened it to public beta on 30 July. It has since become part of LangSmith's bring-your-own-cloud deployments. Braintrust deprecated its earlier AI proxy and pointed users to a new gateway, which is in preview and free until general availability. Comet released the Opik LLM Gateway as a beta endpoint that accepts OpenAI-format requests. On 4 November, Langfuse opened its AI Gateway in public beta, on Langfuse Cloud and for self-hosted deployments; the gateway is MIT-licensed and written in Rust.

MLflow, which began as a machine learning lifecycle tool rather than a tracer, has offered an AI gateway for longer, with budget alerts and limits.

Why did tracing tools and gateways used to be separate?

For the past two years, many teams ran LLM applications with two distinct tools. A gateway sat between the application and the model providers: it held the provider API keys, translated between provider APIs and retried failed calls. LiteLLM, an open-source proxy, became a common choice. A tracing tool such as LangSmith or Langfuse sat beside the application and recorded each model call, tool call and retrieval step, so engineers could inspect what an agent had done.

The division reflected different engineering demands. A gateway has to be fast and support every provider's API. A tracing tool needs a good interface, storage for large payloads and an SDK that wraps application code. Langfuse still documents the two-tool setup, with a LiteLLM Proxy integration that sends every call through the proxy and into Langfuse.

The core limitation of that arrangement: a tracing tool can only record a model call after the fact, while a gateway can stop one. As the source analysis puts it, a gateway can block a model call before it reaches the provider while a tracer can only record it afterward.

Why does a budget need the request path?

A tracer records a call after the provider has answered and the tokens have been paid for. That record supports a spend chart, but it cannot stop the spending. Enforcing a team budget, blocking a model or removing a secret from a prompt all require checking the request before it reaches the provider.

LangChain makes this argument in its launch post. It describes teams combining a separate gateway, a guardrails product and an observability tool, and says preventing problems means enforcing policies at the request layer. Its gateway answers a call that would exceed a budget with HTTP status 402 instead of forwarding it to the provider.

How does one entry point capture untraced calls?

The second driver is coverage. An SDK records only instrumented code paths, and calls outside those paths are where agent failures go unnoticed. A script on a laptop, a notebook or a coding agent using a copied API key leaves no trace. A gateway holding the provider keys sees every call, because no other route to the provider exists.

Langfuse builds its gateway around this principle. According to its gateway documentation, each gateway key belongs to a user or service account and can carry metadata such as a team or cost centre. Only the gateway holds the provider credential, and administrators approve which models each credential may call. LangSmith publishes setup pages for routing Claude Code, Codex and Gemini CLI through its gateway — agents that call models from a developer's machine, outside any application's tracing code.

What does a tracer's gateway add over a standalone one?

The vendors also argue the reverse: a standalone gateway is insufficient alone. Langfuse's launch post says a standalone gateway can only count tokens and cost, while its own gateway makes each call part of a full trace. The Langfuse gateway accepts a W3C traceparent header, so a call through it joins the trace the application has already started. By default it records usage and cost, not prompt and response content.

This is the strongest technical case for consolidation. A standalone gateway sees a flat list of completions. A tracer already holds the agent's tool calls and retrieval steps from its SDK and can place each gateway call inside that structure.

A commercial reading exists, which the vendors do not state themselves. Whoever runs the gateway holds the provider keys and the spend report — the report finance and platform teams read. That makes a gateway harder to replace than a trace viewer. LangSmith and Langfuse already exploit the position differently: LangSmith sells Gateway Credits, letting customers call hosted models such as Kimi without their own provider key, while Langfuse's FAQ states it does not resell tokens.

What can the beta gateways not do yet?

The vendors document their own gaps:

• Langfuse's gateway supports only OpenAI and Anthropic today, with Amazon Bedrock, Google Vertex AI and Google AI Studio on its roadmap.
• Fallbacks between providers and hard spend caps per team, user or key are planned but not shipped; the gateway currently offers a spend dashboard and alerts.
• Langfuse's documentation does not describe rate limits or response caching.
• The gateway does not convert between API formats, so an OpenAI client can call only OpenAI models.
• The self-hosted version calls official provider URLs only, ruling out Azure OpenAI, Bedrock or an internal proxy for now.
• Comet describes the Opik gateway as designed for development and testing.
• Braintrust has not announced gateway pricing.

LangSmith's gateway does more. Its documentation lists fallbacks, spend limits, rate limits and data policies as current features, and it accepts the OpenAI Chat Completions, OpenAI Responses and Anthropic Messages formats. Fallbacks and rate limits were listed as future work in May, so they arrived during the beta. The product remains labelled beta overall.

Load testing exposed an operational gap. A self-hosted benchmark of the Langfuse gateway, LiteLLM and AcruxCore found the Langfuse gateway added less latency per call than LiteLLM. With default Docker settings, however, it failed 739 of 1,000 simultaneous streams, because its process kept Docker's default limit of 1,024 open files. One line in Langfuse's Compose file fixed that.

LiteLLM, by comparison, has supported more than 100 providers for years and already includes virtual keys, spend tracking, guardrails, load balancing and an admin dashboard. It is also moving its request path from Python to Rust: its Rust AI Gateway is in beta, and the team expects Anthropic-style /v1/messages requests to run end to end in Rust by 1 December, with about 20 times less latency. MLflow and AcruxCore already ship a gateway and tracing in one product.

Why might teams keep the two tools apart?

The largest cost of merging is coupling. With a separate gateway, a team changes tracing vendors by changing a logging callback. When the tracer is also the gateway, every service's base URL, keys and model identifiers change too — Langfuse writes model identifiers in the form credential:model.

The tracer also becomes a runtime dependency. A tracing backend outage costs traces; a gateway outage fails model calls. A tracing vendor running the gateway now needs the uptime of critical infrastructure.

Security vendors want the same position. Palo Alto Networks completed its acquisition of Portkey, a standalone gateway, in May 2026 and made it the AI gateway of its Prisma AIRS platform. A security team may want the gateway under its own control rather than inside an observability tool. Gateways have also become attack targets, as the LiteLLM supply-chain breach in March showed.

How should teams choose?

Teams can start by listing what must happen before a model call. Visibility alone requires a tracer beside the request path, with no runtime dependency. Budgets, key policies or model approval require a gateway. Provider coverage comes next: a team should write down every provider and API format used in production, including Azure OpenAI and Bedrock, and compare it against each gateway's current support rather than its roadmap.

Teams already running LiteLLM have little reason to move until a beta gateway offers something they lack. The main addition from a tracer's gateway — a gateway call placed inside the agent trace — has to be worth a migration. Teams starting fresh can keep the gateway as its own layer even when one vendor supplies both, holding the base URL and keys in one configuration module so a later move is a configuration change rather than a code change.

Most of these gateways carry beta or preview labels, so the gap lists will shrink quickly. The open question is ownership of the request path inside a typical company: the observability team, the platform team running a standalone gateway, or the security team. This year LangSmith and Langfuse built gateways of their own and Palo Alto Networks bought one, yet in many stacks the request path still runs through LiteLLM.