How OpenAI Scaled PostgreSQL for Hundreds of Millions of ChatGPT Users

OpenAI’s public interviews and the recent engineering blog post make it clear that the service has moved from handling a few thousand API calls per day in its early beta to processing hundreds of millions of requests daily at production scale. The team repeatedly emphasizes three operational pillars: raw request volume, latency targets, and the choice to retain PostgreSQL as the primary transactional store.

**Request volume** – In a 2023 interview, OpenAI’s infrastructure lead noted that the platform now sustains “tens of thousands of requests per second” across multiple data centers, which aggregates to “well over 200 million requests per day.” This growth has driven a shift from single‑node PostgreSQL instances to a horizontally sharded cluster with automated fail‑over, while still preserving the relational model for billing, usage accounting, and audit trails.

**Latency goals** – The engineering blog outlines two latency envelopes:

• External API latency: ≤ 100 ms for the 99th percentile of completion calls, measured from request receipt to response dispatch.
• Internal routing latency: ≤ 10 ms for service‑to‑service calls that query user quotas or token counters, enforced by colocating PostgreSQL read replicas with the routing layer.

To meet these targets, OpenAI employs a combination of connection pooling (PgBouncer), prepared‑statement caching, and a custom “fast‑path” read‑only replica that serves the most frequent quota checks.

**Why PostgreSQL remains the core store** – The decision was justified on three technical grounds:

• Strong ACID guarantees are required for financial billing and per‑user quota enforcement.
• Rich query capabilities allow complex joins between usage logs, model version tables, and customer metadata without resorting to separate NoSQL layers.
• Maturity of tooling – existing backup, replication, and observability ecosystems reduce operational risk compared to a newly‑built distributed store.

Below is a minimal Go transaction that illustrates how the service records a usage event while ensuring atomicity:

func recordUsage(ctx context.Context, db *pgxpool.Pool, userID string, tokens int) error {
    tx, err := db.BeginTx(ctx, pgx.TxOptions{})
    if err != nil { return err }
    defer tx.Rollback(ctx)

    // Decrement quota
    _, err = tx.Exec(ctx,
        `UPDATE quotas SET remaining = remaining - $1
         WHERE user_id = $2 AND remaining >= $1`,
        tokens, userID)
    if err != nil { return err }

    // Insert audit log
    _, err = tx.Exec(ctx,
        `INSERT INTO usage_log (user_id, tokens, ts)
         VALUES ($1, $2, now())`,
        userID, tokens)
    if err != nil { return err }

    return tx.Commit(ctx)
}

The snippet demonstrates the trade‑off between strict consistency (required for billing) and the need for ultra‑low latency; the transaction is kept deliberately short, and the surrounding architecture (connection pool, read‑only replicas) ensures that the 10 ms internal latency envelope is respected even under peak load.

The system partitions the user base across N PostgreSQL instances using a deterministic hash of the immutable user_id. The hash function is a 64‑bit Murmur3 variant, reduced modulo the shard count, guaranteeing even distribution without requiring a central lookup table. For example, a user with user_id = 123456789 on a 16‑shard deployment maps to shard (murmur3_64(123456789) % 16) = 7. This approach eliminates hot‑spot risk and supports horizontal scaling by adding or removing shards with minimal data movement.

Tenant isolation is achieved at the schema level. Each tenant owns a dedicated schema (e.g., tenant_42) within its assigned shard. All tenant tables share a common logical model but are physically separated, allowing per‑tenant vacuuming, backup, and permissioning. The core tables include:

• users (user_id BIGINT PRIMARY KEY, tenant_id INT, …)
• orders (order_id BIGINT PRIMARY KEY, user_id BIGINT REFERENCES users(user_id), …)
• audit_log (event_ts TIMESTAMP, tenant_id INT, payload JSONB)

Because tenant_id is stored on every row, cross‑tenant queries can be rejected early by the FDW. The custom foreign‑data wrapper (FDW) implements GetForeignRelSize, GetForeignPaths, and GetForeignPlan to intercept planner calls. The routing logic resides in fdw_shard_router.c:

static int
select_shard(Oid relid, Datum user_id)
{
    uint64_t h = murmur3_64(DatumGetInt64(user_id));
    return (int)(h % shard_count);
}

/* Planner hook */
static void
fdw_explain_foreign_scan(ExplainState *es, ForeignScan *fs)
{
    int shard = select_shard(fs->relid, fs->fdw_private[USER_ID_IDX]);
    appendStringInfo(es->str, "Routing to shard %d", shard);
}

During execution, BeginForeignScan opens a connection to the target shard using a pre‑configured connection pool, then forwards the original SQL with tenant‑scoped SET search_path = tenant_XX. This ensures that all subsequent operations run inside the correct tenant schema without additional query rewriting.

Trade‑offs:

• Pros: Stateless routing, linear scalability, strong tenant data isolation.
• Cons: Rebalancing requires data migration; cross‑shard joins must be materialized in the application layer or via a distributed query engine.

OpenAI’s high‑throughput API services place a pgBouncer instance directly in front of each PostgreSQL cluster, operating exclusively in transaction‑pooling mode. In this mode a client connection is bound to a physical server socket only for the lifetime of a single SQL transaction; once the transaction commits or rolls back, the socket is released back to the pool. This eliminates the “one‑connection‑per‑thread” pattern that would otherwise exhaust the database’s max_connections limit under bursty traffic.

The broker is configured with a minimal static pool size and a dynamic scaling layer written in Go that runs alongside the API workers. The algorithm samples three metrics every 100 ms:

• active_tx – number of transactions currently holding a socket.
• queue_len – length of the client request queue waiting for a socket.
• latency_ms – average time from request receipt to transaction start.

Based on these inputs it computes a target pool size target = base + α·queue_len + β·latency_ms, where α and β are tunable coefficients. The pool is then expanded or contracted by issuing SET pool_size = N via the SHOW POOLS admin console, without restarting pgBouncer.

# Example dynamic resize loop (simplified)
for {
    stats := fetchMetrics()
    target := baseSize + alpha*stats.queueLen + beta*stats.latencyMs
    if target != currentSize {
        exec("SET pool_size = %d", target)
        currentSize = target
    }
    time.Sleep(100 * time.Millisecond)
}

When the computed target approaches the hard limit set by max_client_conn, OpenAI’s fallback path activates:

• New incoming HTTP requests are placed in an in‑memory throttling queue; if the queue exceeds a configurable threshold, the service returns 429 Too Many Requests.
• A secondary “overflow” pgBouncer instance, pre‑seeded with a smaller static pool, is spun up on demand to absorb excess traffic.
• If both brokers are saturated, the API layer switches to a read‑only replica for non‑mutating queries, preserving availability at the cost of eventual consistency.

Trade‑offs of this architecture include:

• Pros: Near‑linear scaling of concurrent transactions, reduced memory footprint per worker, and graceful degradation under load.
• Cons: Transaction‑pooling disallows session‑level state (e.g., temporary tables) and requires careful query design; dynamic resizing adds a small control‑plane latency.

By tightly coupling pgBouncer transaction‑pooling with a responsive pool‑size controller and a multi‑tier fallback strategy, OpenAI can sustain massive concurrency while keeping PostgreSQL connection counts well within safe operating bounds.

The deployment consists of three primary clusters—North America (us‑east‑1), Europe (eu‑central‑1), and APAC (ap‑southeast‑2)—each exposing a read‑replica mesh that can serve read traffic locally while asynchronously replicating write‑ahead logs (WAL) to the other regions. Each primary runs PostgreSQL 15 with max_wal_senders=10 and wal_level=logical, enabling both physical streaming and logical replication slots.

Physical streaming replication is configured with a dedicated network interface and TLS‑encrypted connections. The primary_conninfo string on each replica includes:

primary_conninfo = 'host=primary-us-east-1.example.com port=5432 
    user=replicator password=******** sslmode=require 
    application_name=replica-eu-central-1'

Key WAL shipping parameters are tuned to balance latency and bandwidth:

• wal_sender_timeout = 60s – ensures a stalled sender is reclaimed quickly.
• wal_receiver_status_interval = 10s – provides timely feedback for flow control.
• max_replication_slots = 6 – three physical slots (one per remote region) plus three logical slots for feature‑flag streams.
• wal_compression = on – reduces cross‑continent traffic by ~30 % in typical workloads.
• wal_keep_segments = 1000 – enough buffer to survive brief network outages without triggering a full base backup.

Logical replication is used for feature‑flag rollouts that require row‑level filtering. A publication is created on the primary:

CREATE PUBLICATION feature_flags_pub 
    FOR TABLE feature_flags 
    WHERE environment = 'beta';

Each region subscribes with a dedicated slot, allowing the flag table to be streamed without affecting the bulk data replication path:

CREATE SUBSCRIPTION beta_flags_sub 
    CONNECTION 'host=primary-us-east-1.example.com dbname=app 
    user=replicator password=********' 
    PUBLICATION feature_flags_pub;

Trade‑offs of this mesh topology include:

• Pros: Read latency is minimized by local replicas; failover can be performed regionally without global coordination.
• Cons: Write latency increases due to cross‑region WAL propagation; network partitions can cause replica lag that must be monitored with pg_stat_replication.
• Operational complexity: Managing multiple replication slots and ensuring slot retention policies do not exhaust disk space.

Monitoring scripts typically query pg_stat_replication and pg_replication_slots every 30 seconds, alerting when write_lag exceeds 200 ms or when slot disk usage surpasses 80 % of the allocated WAL volume.

Observability in a distributed PostgreSQL‑centric stack is built on three pillars: time‑series metrics, distributed tracing, and deterministic recovery. The metrics pipeline uses Prometheus exporters embedded in each service and the postgres_exporter for database‑level counters. A typical scrape configuration isolates per‑node metrics and aggregates them at the central Prometheus server:

scrape_configs:
  - job_name: 'postgres'
    static_configs:
      - targets: ['db-primary:9187', 'db-replica-1:9187', 'db-replica-2:9187']
    relabel_configs:
      - source_labels: [__address__]
        regex: '(.*):9187'
        target_label: instance
        replacement: '${1}'

Key metric groups include:

• pg_stat_activity_count – active sessions per database.
• pg_replication_lag_seconds – lag of each replica as reported by pg_stat_replication.
• pg_auto_failover_state – current role (primary/secondary) and health status.
• Application‑level latency histograms (http_request_duration_seconds) exported via OpenTelemetry SDKs.

Tracing is instrumented with OpenTelemetry, propagating traceparent headers across HTTP, gRPC, and SQL drivers. The following Go snippet shows how a database call is wrapped:

ctx, span := tracer.Start(ctx, "SELECT users")
defer span.End()
rows, err := db.QueryContext(ctx, "SELECT id, name FROM users WHERE id = $1", userID)
if err != nil {
    span.RecordError(err)
    span.SetStatus(codes.Error, err.Error())
}

Trace data is exported to a Jaeger collector, enabling end‑to‑end latency analysis and root‑cause isolation when a replica falls behind or a query spikes.

Automated recovery leverages pg_auto_failover with a predefined failover playbook. The coordinator monitors health checks (heartbeat interval = 2 s, timeout = 6 s) and triggers a role swap when the primary’s pg_auto_failover_state transitions to down. The playbook executes the following steps:

• Pause traffic at the load balancer (e.g., set drain flag on primary endpoint).
• Run pg_autoctl perform failover --node-id=<primary_id> to promote the most up‑to‑date secondary.
• Update DNS or service discovery records to point to the new primary.
• Resume traffic after health checks confirm pg_stat_replication lag < 100 ms.

Trade‑offs include the latency introduced by health‑check intervals versus detection speed, and the need to keep application connection pools aware of role changes to avoid transient “no‑primary” errors. Properly calibrated thresholds and idempotent deployment scripts mitigate these risks while preserving automated resilience.

OpenAI’s publicly released post‑mortems reveal three recurring production themes that directly affect large‑scale inference services: the hidden cost of over‑sharding, the latency impact of replica lag, and the operational tension between strong consistency and high availability.

Over‑sharding – In the “ChatGPT latency spike” incident, the team increased the shard count from 64 to 256 to accommodate a sudden traffic surge. While the intent was to spread load, each additional shard introduced a new network hop and a per‑shard warm‑up cost of ~15 ms. The cumulative effect pushed average request latency beyond the 200 ms SLO. The post‑mortem recommends a “shard‑budget” policy that caps the number of shards per service tier and enforces a cost‑benefit test before scaling.

• Measure per‑shard overhead (network, serialization, GC) before adding shards.
• Prefer horizontal scaling of existing shards (adding nodes) over increasing shard count.
• Automate rollback of shard‑count changes if latency SLOs degrade >5 %.

Replica lag – A separate outage was traced to a lagging read‑replica in a multi‑region PostgreSQL cluster. The replica fell behind by ~5 minutes, causing stale model‑parameter reads that added ~120 ms to end‑to‑end latency and violated the 99th‑percentile latency SLO. The fix involved tightening the replication timeout and adding a health‑check that routes traffic away from lagging replicas.

# Example health‑check snippet (Python)
def is_replica_healthy(replica):
    lag = replica.get_replication_lag_seconds()
    return lag < 30  # seconds

Consistency vs. availability – OpenAI’s “model‑parameter sync” incident highlighted the trade‑off: enforcing strict linearizability across data centers increased cross‑region round‑trip time by ~80 ms, while a relaxed eventual consistency model kept latency within SLO but risked brief parameter divergence. The team adopted a hybrid approach: critical configuration writes use a quorum (N/2 + 1) to guarantee consistency, whereas bulk model weight updates use asynchronous propagation with versioned checkpoints.

• Classify data by consistency criticality.
• Use quorum reads/writes for high‑criticality items.
• Monitor divergence metrics and trigger fallback to strong consistency when drift exceeds thresholds.

These lessons underscore that scaling decisions must be evaluated against measurable latency budgets, that replica health is a first‑order SLO driver, and that a nuanced consistency model can preserve both performance and correctness in globally distributed AI services.