# Cloudflare Saved 100 TB of RAM With High-School Statistics. Here's the Lesson for Your Node Backend

---

Last week Cloudflare published a post about cutting **roughly 100 TB of RAM** out of their internal load balancer. They didn't do it with a rewrite or new hardware. They counted how many hash points their consistent-hashing rings actually needed, found out it was about 10% of what they were storing, and deleted the other 90%.

I've copied "use 160 virtual nodes per server" into more than one project without ever asking where 160 came from. If you've ever sharded a Redis cache, spread uploads across storage nodes, or pinned WebSocket users to a Node process, you've probably done the same thing. So let's go through what Cloudflare did and rebuild a small version in Node.

## The setup: consistent hashing in 60 seconds

Cloudflare's Pingora Backend Router sends cacheable requests to cache servers by hashing the URL, so each file lives on exactly one machine per data center. A plain `hash(url) % servers.length` breaks as soon as a server joins or leaves, because almost every key moves and your cache hit rate drops to zero.

Consistent hashing fixes this. Picture the 32-bit hash space as a ring. Each server gets hashed onto the ring, every key gets hashed onto it too, and a key belongs to the first server it meets going clockwise. Remove a server and only *its* keys move.

With one point per server, though, the load is badly uneven. Cloudflare's numbers: with 100 servers and one point each, the coefficient of variation (standard deviation divided by mean) is about **99%**. Some servers get twice their share while others sit idle. The usual fix is to give each server lots of "virtual" points. NGINX hardcodes 160, Pingora used the same default, and that drops the error to around 8%.

![](https://cdn.hashnode.com/uploads/covers/69d007f5e466e2b7625cd1df/165ad0a0-2907-4d60-b75e-910c8e09484b.png align="center")

## Where 100 TB came from

Two things made the memory grow.

**Weights.** Cloudflare uses the Ketama scheme, where a server with more disk gets proportionally more points. A server with weight 625 on a base of 160 ends up with **100,000 points**.

**Multiple rings.** Because of compliance rules and per-customer caching features, not every server can serve every request. You can't express "only these servers" by adding points to one ring, so every combination of features gets its own ring. A handful of features turns into dozens of rings, and some processes were holding **~6 GB** of hashing data.

Run that across every data center they operate and you get the 100 TB.

## The math that let them delete 90% of it

The useful part of the post is a formula. With N servers and k points each, the error is:

```plaintext
CV(k) = √( (N − 1) / (N·k + 1) )
```

Because k sits under a square root, cutting the error by 10x costs you about 100x more points. Cloudflare's example: the final 90,000 hashes on that weight-625 server improved distribution by only **~0.7%**. And past a certain point extra points make things *worse*: 32-bit hashes start colliding (the birthday paradox), and each collision quietly removes a point's contribution. In their 2,048-server simulations, error actually went up somewhere between 10k and 100k hashes per server.

So they cut the hash count by about 90% without any noticeable loss in balance.

You don't have to take their word for it. Here's a compact ring in Node that you can run yourself:

```javascript
import { createHash } from "node:crypto";

const hash32 = (s) => createHash("md5").update(s).digest().readUInt32BE(0);

export class HashRing {
  constructor(servers, pointsPerServer = 160) {
    const total = servers.length * pointsPerServer;
    this.servers = servers;
    // 6 bytes per point: 4 for the hash, 2 for the server index
    this.hashes = new Uint32Array(total);
    this.index = new Uint16Array(total);

    const tmp = [];
    servers.forEach((name, i) => {
      for (let p = 0; p < pointsPerServer; p++) tmp.push([hash32(`${name}-${p}`), i]);
    });
    tmp.sort((a, b) => a[0] - b[0]);
    tmp.forEach(([h, i], j) => { this.hashes[j] = h; this.index[j] = i; });
  }

  get(key) {
    const h = hash32(key);
    let lo = 0, hi = this.hashes.length;
    while (lo < hi) {                       // binary search: first point >= h
      const mid = (lo + hi) >>> 1;
      if (this.hashes[mid] < h) lo = mid + 1; else hi = mid;
    }
    return this.servers[this.index[lo % this.hashes.length]];
  }
}
```

I pushed 200,000 fake asset URLs through it with 100 servers and compared the measured spread to the formula:

| Points per server | Measured CV | Formula |
| --- | --- | --- |
| 1 | 106.6% | 99.0% |
| 10 | 34.7% | 31.4% |
| 40 | 16.3% | 15.7% |
| 160 | 7.4% | 7.9% |
| 1000 | 4.0% | 3.1% |

(At 1000 points the measured number is mostly sampling noise from only having ~2,000 requests per server.) Look at how the curve behaves: going from 1 to 40 points is a huge improvement, and going from 160 to 1000 barely changes anything while using 6x the memory.

![](https://cdn.hashnode.com/uploads/covers/69d007f5e466e2b7625cd1df/1e8ece5e-171c-4d89-9423-396109e80d49.png align="center")

## The 2-byte trick that saved another 25%

This is my favorite detail. Each ring entry was a Rust struct with a `u32` hash and a `u32` server index, so 8 bytes. Since a server index never needs more than 65k values, they changed the index to `u16`... and saved nothing. Alignment rules pad the struct back up to 8 bytes.

What actually worked was storing each point as a raw `[u8; 6]` and reading the fields through getters. That one change cut ring memory by **25%**.

The JavaScript version of this lesson is in the class above. If you store the ring as an array of `{ hash, server }` objects, every entry pays for a V8 object header, a hidden-class pointer, and a slot in the array. Two parallel typed arrays (`Uint32Array` + `Uint16Array`) cost exactly 6 bytes per point, sit in contiguous memory, and are friendlier to the garbage collector. It's the same idea: lay out your data the way the machine stores it.

The same applies in Python. A list of tuples for a 10-million-point ring will eat gigabytes, while two NumPy arrays (`np.uint32`, `np.uint16`) plus `np.searchsorted` give you the same lookup in a fraction of the space. Keep that in mind next time you shard embeddings or route inference requests across GPU workers.

## Shipping it without nuking the cache

Changing the ring moves keys, and moving keys means cache misses that hit your origin all at once. Cloudflare's rollout is a good template for any "this changes routing" change:

- They kept **both rings in memory** and chose between them per request using a stable hash, which also gave them instant rollback without a redeploy.
- They rolled out **per data center**, not by a global percentage, so problems stayed contained.
- They watched specific signals: ring-version counters, connection errors, process memory, startup time, cache hit rate and origin traffic.
- Only after reaching 100% did they delete the old code path.

If you're moving a Redis cluster from 4 to 6 nodes behind an Express API, this is the same plan at a smaller scale: dual-read, ramp up slowly, watch your database load, then remove the old ring.

## What to take back to your own stack

1. **Question your magic numbers.** 160 is a reasonable default, but whether it's right depends on your N and how much error you can tolerate. Run the CV formula before copying a constant.
2. **Measure memory per entry.** Typed arrays in Node and NumPy dtypes in Python routinely give you 3–10x savings on large lookup tables.
3. **Watch for combinatorial growth.** "One structure per feature combination" is where memory blowups usually hide. Feature flags × regions × tenants grows fast.
4. **Routing changes are cache migrations.** Run them side by side, roll them out in stages, and watch origin load.

## Wrapping up

Of all of this, the error formula is what I'd keep. The Rust and the scale are impressive, but the actual win was some engineers checking how much accuracy they needed and noticing they were paying a lot for precision nobody would ever notice. That happens in plenty of MERN backends too.

Try running the ring above with your own server count and see where the curve flattens for you. If you've done consistent hashing in production with Redis, Mongo, or something you built yourself, I'd like to hear how you picked your numbers. Leave a comment.