OpenAI's Habitat storage layer hits 20M QPS on Cosmos DB

OpenAI's largest Azure Cosmos DB account stores tens of petabytes and serves more than 20 million queries per second, a scale the company says no other organization has deployed. The figure anchors a detailed engineering post, the second in a two-part series, describing how OpenAI scaled Habitat — its storage platform that responds in milliseconds to over 1 billion ChatGPT users across 40 geographic regions and now holds over 500 petabytes of data.

The account matters beyond OpenAI's engineering blog. It documents what happens when one of the world's largest AI companies outgrows a flagship managed cloud database: custom fixes negotiated directly with the provider, two serious outages traced to untested failover behavior, and ultimately the decision to build an in-house exabyte-scale replacement. For infrastructure teams watching AI workloads strain cloud primitives, the post is a case study in the operational costs of frontier scale.

Why did OpenAI abandon single-tenant containers?

Habitat's history follows a circular path. It began with one shared multi-tenant Azure Cosmos DB container, moved to provisioning single-tenant backing containers for isolation, and then reverted to a shared container as the default. Each step responded to concrete failure modes.

In Azure Cosmos DB, an account defines containers, each provisioned with throughput in request units per second (RU/s); data is sharded by logical partition key, and logical partitions sit on physical partitions where limits are actually enforced. OpenAI initially allowed any client load, relying on the documented 10,000 RU/s limit per logical partition key for back pressure. The design broke down in three ways.

First, early API choices exposed partition key selection directly to clients, causing frequent collisions on keys such as user_id or workspace_id across product use cases. Shared limits on the same logical keys constrained per-user throughput. Second, the documented 10,000 RU/s was unreachable in practice: logical partitions share the physical partition's limits with every colocated key, so noisy neighbors disrupted unrelated workloads — niche product features interfered with core login and signup flows. OpenAI had seen the same pattern scaling Postgres before Habitat, where one hot organization's traffic could take out the whole database.

Third, throughput per physical partition shrinks as data grows. Container throughput is distributed roughly evenly across physical partitions, and partitions split as storage thresholds are crossed while total throughput stays fixed. For storage-heavy, blob-like use cases, storage growth alone reduced throughput capacity even with stable request rates.

Separate containers solved part of the problem. Each container receives its own physical partitions, isolating noisy neighbors and partition key space. OpenAI provisioned dedicated containers for critical new use cases and protected legacy services stuck in the shared default container without a data migration. The strategy, in OpenAI's words, "saved us from many site-wide outages, limiting the blast radius to feature or service degradation outages."

What broke the isolation strategy?

Dedicated containers reintroduced the core cost of single-tenancy: capacity planning shifted to product teams. A new use case going from 0 to 1 million queries per second on an isolated container might represent 1,000x more load than the dogfooding baseline. Teams shipped features without pre-scaling, and OpenAI engineers, in their words, would "sit in SEV rooms on launch day attempting to scale up containers last minute." Monitoring usage demanded intimate product knowledge of traffic shifts, and with no central planning, many teams planned poorly.

The company moved back to multi-tenancy by co-engineering fixes with the Azure team:

• Throttling the hottest keys first, preventing one hot key from consuming a physical partition's throughput and acquiring a site-wide blast radius.
• Request tiering QoS, which assigns lower priority to offline workloads such as bulk inserts and data extracts, throttling them before interactive requests that serve end users.
• An improved serverless mode, built for OpenAI, letting every physical partition scale to 10K RU/s and adding RU-usage-based partition splits. The existing serverless mode capped partitions at 5K RU/s and scaled on storage alone, blocking RU-heavy use cases.

Since the shift, OpenAI reports zero provisioning or noisy-neighbor incidents over multiple months, despite accelerating product development — while product teams no longer perform upfront capacity planning.

How serious were the outages that forced co-engineering?

In late 2024, a power outage struck the primary Azure region for one of OpenAI's largest Cosmos DB accounts. The failover revealed that its largest accounts would take several hours to move to a secondary region — a scenario neither OpenAI nor Azure had tested at that scale. Roughly a month later, failing those accounts back triggered a second serious outage: a connection stampede overwhelmed the original region's Cosmos DB gateway because a quirk in the gateway-mode client made all clients reconnect immediately, even though only a subset of physical partitions had returned.

The incidents exposed what OpenAI calls a systemic issue: at its scale, "managed services become less managed." Today the Habitat and Azure teams jointly monitor every layer from Habitat's service tier down to Cosmos DB's physical storage, test against a production-sized non-production account, iterate on SDKs together, and can page each other directly without support tiers. Joint work over the past year added gateway metadata caching to protect Cosmos DB's metadata partition, connection pooling to reduce pressure from the Python habitat-service fleet, workload colocation, and isolation of critical from offline workloads via separate VPCs.

How did OpenAI cut read latency 12x?

With reliability stabilized in 2025, latency became the constraint. Reads represent over 90% of Habitat's requests, and OpenAI pursued a layered strategy:

• Replication: latency-critical datasets such as authentication metadata use highly replicated Cosmos DB accounts — some replicate to all 40 Azure regions, others to at least one region per continent — with a transparent data movement and routing layer built earlier than planned. The approach is capacity-constrained and limited to a few hand-selected datasets.
• Distributed caching: a lookaside caching layer on Valkey servers stores recently accessed objects and edges. Instead of TTL or explicit invalidation, both of which OpenAI describes as error-prone tradeoffs, Habitat follows Meta's TAO model: change data capture (CDC) via Cosmos DB's change feed propagates invalidations globally within seconds, keeping hit rates high and staleness low.
• Rust rewrite: the Habitat service itself was rewritten from Python to Rust to remove per-request overhead.

Combined, the changes cut global read p50 latency from roughly 120 ms to roughly 10 ms — a 12x improvement since the performance push began in 2026.

The CDC platform behind invalidation is itself a major system. Millions of captured changes per second flow across thousands of containers and millions of container partitions, through more than a hundred Kafka topics, powering secondary indexes, data-movement pipelines, and analytics. OpenAI hardened worker lifecycle and lease handling, isolated stream failures, used dead letter queues to prevent head-of-line blocking, tolerated regional outages by switching reader regions, and modified Azure's CDC SDK to batch checkpoints in memory with periodic flushes.

What comes after Cosmos DB?

To scale conversation storage for the next billion users, OpenAI is building a custom exabyte-scale online storage system that combines the RocksDB Cloud core from Rockset with Raft-based replication. The in-development system implements Cosmos DB's document model but uses disaggregated storage and physical replication to cut flash capacity requirements — accepting lower peak QPS in exchange.

The message to cloud providers is embedded in the post's conclusion: any system large or critical enough, OpenAI writes, necessitates breaking down ownership walls and operating across organizational boundaries.