OpenAI Habitat Storage: Cosmos DB Scale and Latency Lessons

OpenAI published the second part of its Habitat storage-platform account on October 23, 2026. The company says Habitat supports product data access for more than one billion users across 40 geographic regions and stores more than 500 petabytes. Its central engineering story is not simply “use a larger database.” OpenAI describes moving from a shared Azure Cosmos DB container, toward more single-tenant isolation, and then back to multi-tenancy by default after adding controls for hot keys, quality of service and data movement. These figures and outcomes are OpenAI’s own operational report; no independent audit is supplied.

The reported 12× improvement combines several changes

OpenAI says global median read latency fell from roughly 120 milliseconds to roughly 10 milliseconds after three broad changes: replicating selected latency-sensitive data nearer users, adding an integrated regional cache and rewriting proxy code in Rust. The article says reads make up more than 90% of Habitat requests. That makes read-path optimization important, but the p50 figure describes the middle request in OpenAI’s system—not tail latency, every ChatGPT action or end-to-end response time.

Canadian users should not treat the number as a measurement of their ISP or device. A ChatGPT interaction can involve model inference, application logic, safety systems and several remote services in addition to storage. An internet speed test measures another path again. The useful takeaway is methodological: state exactly which layer and percentile improved.

Caching improves speed but creates a freshness problem

Habitat uses Azure Cosmos DB change feeds to push cache invalidations to regional Valkey caches. OpenAI says this provides high hit rates while limiting stale data, with time-to-live expiry remaining as a backstop. The report also describes failure isolation, dead-letter queues and alternate-region readers for change-data-capture pipelines.

These mechanisms reduce repeated long-distance reads, but they do not eliminate the design tradeoff between freshness, availability and latency. Teams adopting a similar pattern should test lost or delayed invalidations, regional failure and the slow uncached path rather than copying the headline median. Microsoft’s Cosmos DB guidance also emphasizes choosing partition keys that distribute data and workload evenly.

Scale claims belong to this deployment

OpenAI reports that its largest Cosmos DB account stores tens of petabytes and serves more than 20 million queries per second. It also says power, cooling and regional capacity can limit where data is placed. Those constraints explain why only small, latency-sensitive datasets are broadly replicated.

This is a completed internal engineering account with forward-looking elements: OpenAI says a custom exabyte-scale system remains under development. Report the production Cosmos DB work separately from that future system. For Canadian organizations, the transferable questions are which data must remain in-region, which reads tolerate staleness, and how failover behaves at production scale.

Sources and editorial method

We use the linked original statements and reports to distinguish documented facts from our analysis. Company announcements describe their authors’ plans and claims; they are not independent verification of future outcomes. This article is dated and will be revised if material evidence changes. Read our editorial standards and corrections process.

OpenAI: scaling Habitat storage, part II ↗ — October 23 engineering account; architecture, scale, latency claims and future work.
Microsoft Learn: partitioning and horizontal scaling in Azure Cosmos DB ↗ — Official background on logical partitions, physical partitions and partition-key design.