OpenAI's Commits Grew Almost 15x. Git Was Fine, the Server Wasn't

---

Most scaling war stories start with a traffic spike. OpenAI's starts with its own coding agent: one engineer now has several changes ready for CI at once, and each push can kick off several jobs fetching the same commit.

Yesterday OpenAI published how it rebuilt the backend behind its monorepo. The new system is called Pando, it's internal, and you can't install it. I still think it's the most useful system design read of the month, because the ideas fit in a MERN backend you already run.

What actually broke

The numbers are a bit silly. Commits landing on master grew 1,378% between Q1 2024 and Q3 2026, which is almost 15x, and OpenAI's own estimate for Q4 is 2,442%. Reliability on the old backend at times fell below 90%, with slow fetches and failed merges.

Here's the part I liked: they kept Git. Same git push, same git fetch, same wire protocol, so editors, scripts and CI didn't need to change. The New Stack's write-up puts the conclusion well: the problem was how conventional Git servers are built, not the client or the protocol.

Three things were hurting:

• Pack generation was CPU-heavy, and the same pack got rebuilt for every client asking for it
• Ref updates were serialized across the whole repo
• The branch list itself got huge (with millions of branches, advertising them all can exceed 100 MiB)

Swap "ref" for "document" and "pack" for "report PDF" and that could be your SaaS.

Lesson 1: stop making unrelated writes wait for each other

OpenAI modeled the two textbook designs, three-phase commit across replicas and a write-ahead log with a shared manifest. Both still made pushes to unrelated branches wait on a shared point. So Pando only serializes pushes to the same ref. Everything else commits concurrently.

You can feel the difference with a toy. This runs 200 pushes across 50 branches, with a fake 5ms metadata write:

// ref-lock.mjs
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));

class KeyedQueue {
  tails = new Map();
  run(key, fn) {
    const prev = this.tails.get(key) ?? Promise.resolve();
    const next = prev.then(fn, fn);
    this.tails.set(key, next);
    next.finally(() => {
      if (this.tails.get(key) === next) this.tails.delete(key);
    }).catch(() => {});
    return next;
  }
}

async function simulate(label, keyFor) {
  const queue = new KeyedQueue();
  const latencies = [];
  const start = performance.now();
  const pushes = Array.from({ length: 200 }, (_, i) => {
    const branch = `agent/${i % 50}`;
    const t0 = performance.now();
    return queue.run(keyFor(branch), async () => {
      await sleep(5); // pretend this is the metadata write
      latencies.push(performance.now() - t0);
    });
  });
  await Promise.all(pushes);
  latencies.sort((a, b) => a - b);
  console.log(
    `${label}: total ${Math.round(performance.now() - start)}ms, ` +
      `median push ${Math.round(latencies[100])}ms`
  );
}

await simulate("one repo-wide lock", () => "repo");
await simulate("one lock per branch", (branch) => branch);

node ref-lock.mjs

On my run the repo-wide lock took about 1,050ms with a median push around 530ms. Per-branch took about 22ms with a median around 15ms. It's a queueing demo, not a benchmark, but notice that nothing got faster. The work just stopped waiting in line.

That queue only lives in one Node process, though. Once you have several instances, push the guarantee down to the database. MongoDB updates a single document atomically, so a version check gives you per-ref serialization for free:

// refs.mjs
export class RefConflict extends Error {}

export async function moveRef(refs, name, expectedVersion, newTip) {
  const updated = await refs.findOneAndUpdate(
    { _id: name, version: expectedVersion },
    { $set: { tip: newTip, updatedAt: new Date() }, $inc: { version: 1 } },
    { returnDocument: "after" }
  );
  if (!updated) throw new RefConflict(`${name} moved, fetch and retry`);
  return updated;
}

I fired 10 concurrent writers at one ref and got 1 winner and 9 conflicts. With 10 writers on 10 different refs, all 10 won. Those conflicts are basically a rejected non-fast-forward push: fetch, rebase, try again. Map them to a 409 in your Express route and let the client decide.

Lesson 2: store it once, then hand out links

The old flow generated packs per request. In Pando, every push stores its pack once in blob storage, and readers get links to the packs they need via Git's packfile-uri feature.

The results are the kind you screenshot: nearly a petabyte of Git data a day from a handful of CPU cores, with cache hit rates over 99%. (The delivery service is called Express. No relation.)

If your Node process is still piping big files out of res, here's the same move:

// packs.mjs
import express from "express";
import { S3Client, GetObjectCommand } from "@aws-sdk/client-s3";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";

const app = express();
const s3 = new S3Client({ region: process.env.AWS_REGION ?? "us-east-1" });
const BUCKET = process.env.BUILDS_BUCKET ?? "my-build-artifacts";

app.get("/artifacts/:sha256", async (req, res, next) => {
  const { sha256 } = req.params;
  if (!/^[a-f0-9]{64}$/.test(sha256)) return res.sendStatus(400);
  try {
    const url = await getSignedUrl(
      s3,
      new GetObjectCommand({ Bucket: BUCKET, Key: `artifacts/${sha256}` }),
      { expiresIn: 300 }
    );
    res.redirect(302, url);
  } catch (err) {
    next(err);
  }
});

app.listen(3000, () => console.log("listening on :3000"));

npm install express @aws-sdk/client-s3 @aws-sdk/s3-request-presigner

Keying by content hash is what makes this safe. Those objects never change, so no cache can ever serve you a stale one. Your event loop goes back to doing auth and metadata.

Lesson 3: write small, compact later

Every push makes a small L0 pack, and background workers merge those into bigger "covers", up to L4. If that sounds like RocksDB or LevelDB, that's because it's the same LSM-tree idea. Writes stay cheap because the median pack is under 20 KiB, and the p99 is under 3 MiB.

Without compaction, a fresh clone would pay for all that cheap writing. A quick Python sketch shows how much, assuming we merge every 8 packs (my number, not OpenAI's):

# compaction.py
FANOUT = 8  # merge 8 packs at one level into 1 at the next
MAX_LEVEL = 4


def packs_for_clone(pushes, compact=True):
    levels = [0] * (MAX_LEVEL + 1)
    for _ in range(pushes):
        levels[0] += 1
        if not compact:
            continue
        for lvl in range(MAX_LEVEL):
            if levels[lvl] == FANOUT:
                levels[lvl] = 0
                levels[lvl + 1] += 1
    return sum(levels)


for pushes in (1_000, 100_000):
    print(
        f"{pushes:>7} pushes: {packs_for_clone(pushes, False):>6} packs raw, "
        f"{packs_for_clone(pushes):>3} after compaction"
    )

python3 compaction.py

   1000 pushes:   1000 packs raw,  13 after compaction
 100000 pushes: 100000 packs raw,  33 after compaction

100,000 files down to 33. Same pattern works for an append-only events collection you roll up into hourly and daily summaries, or small AI embedding batches you merge into bigger index segments.

Lesson 4: give up the guarantee nobody uses

This is the one that takes nerve. Pando dropped repo-wide ordering and settled on causal consistency: if you fetch a branch, you get the history of whoever wrote it. Two branches read together might not describe one single moment in the repo. OpenAI's bet is that people and agents care about the order of updates to one ref, not a global order.

MongoDB has this exact model built in, and it's underused. It needs a replica set:

const session = client.startSession({ causalConsistency: true });
try {
  await refs.updateOne(
    { _id: name },
    { $set: { tip: newTip } },
    { session, writeConcern: { w: "majority" } }
  );
  const ref = await refs.findOne(
    { _id: name },
    { session, readPreference: "secondary", readConcern: { level: "majority" } }
  );
  console.log(ref.tip); // always sees newTip, even on a lagging secondary
} finally {
  await session.endSession();
}

That's read-your-own-writes from a secondary, without sending all your reads to the primary.

What I'd actually do with this on Monday

• List your global locks and shared queues, then ask which ones protect anything truly global. Most are per-tenant or per-document in disguise.
• Find endpoints that regenerate the same output per request. Hash it, store it once, redirect.
• Stop advertising everything. Pando only advertises trunk by default, and your GET /everything probably deserves the same treatment.
• Plan for agents as a traffic source. They retry, they fan out, and they don't sleep. Rate limits sized for humans won't hold.

One caveat before you go rewrite things: this took a team with deep Git expertise and a 15x problem. If you've got one instance and 40 users, a single lock is great. It's simple, and you can delete it the day it hurts.

Wrapping up

OpenAI went from kickoff to production in 10 weeks, ran Pando as a proxy in front of the old system so traffic moved without downtime, and got median push latency down nearly 5x. None of the ideas are new, and that's sort of the point: per-key serialization, immutable blobs, LSM compaction and causal consistency have been around for years. Agents just made them urgent sooner.

So go look at your hottest write path. What's it waiting on, and does it really need to? Tell me what you find in the comments.