Right now, I’m working on an InfiniBand topology design for a GPU cluster. The math keeps pointing to the same conclusion: scale-out only makes sense when scale-in has topped out.

It’s not about CUDA cores. It’s not about tensor throughput. It’s about tail latency.

NVLink keeps GPU-to-GPU communication on-package or over short copper links — no NIC, no PCIe host traversal, no protocol stack. For small messages, that means sub-microsecond latency in the hundreds-of-nanoseconds range. InfiniBand NDR switches add sub-microsecond port-to-port latency, but once you include the full path — PCIe to the NIC, driver overhead, fabric hops, and back — real-world GPU-to-GPU latency across nodes often lands in the 3-10μs range depending on message size and topology.

That gap compounds through every collective operation. For communication-heavy workloads, a GB200 NVL72 rack where 72 GPUs share a single NVLink domain will often outperform a 72-GPU InfiniBand cluster — not because of compute, but because tighter integration eliminates the overhead that dominates at small message sizes.

I learned this lesson twenty years ago. Apparently I needed to learn it again.

The Beowulf Lesson (2006)

In 2006, I spent months deploying a Beowulf cluster. Over 200 CPUs, commodity hardware, Linux, MPI — the whole playbook for “HPC on a budget.”

The goal was scale-out compute. The theory was simple: more nodes, more throughput. Distribute the work, collect the results, win.

The reality was different.

After months of tuning, the most effective way to improve performance wasn’t adding more nodes. It was scaling in — putting more compute on fewer machines. The cluster’s 1 Gbps interconnect wasn’t fast enough to keep 200 CPUs fed. Nodes spent more time waiting for data than processing it.

Tail latency commanded the entire game.

The Hierarchy Nobody Escapes (2015)

In 2015, I saw Chad Sakac — then in a senior role at EMC — give a presentation that crystallized what I’d learned the hard way. He walked through the x86 latency hierarchy, showing how each hop away from the processor adds roughly an order of magnitude in delay:

The insight isn’t complicated: your system runs at the speed of its slowest component in the critical path.

You don’t have a “fast system” with one slow layer. You have a slow system with some fast parts that don’t matter.

Why My Beowulf Cluster Failed

Back to 2006. My 200-CPU cluster had:

• Fast CPUs (for the era)
• Enough RAM per node
• Local storage that was adequate
• A 1 Gbps interconnect

That interconnect was the entire problem.

At 1 Gbps, moving data between nodes took microseconds to milliseconds. Meanwhile, CPUs could execute millions of operations per millisecond. The ratio was brutal — for every useful computation, nodes spent orders of magnitude more time waiting for network transfers.

Worse: MPI collectives amplify the slowest node.

In a distributed computation, when all nodes need to synchronize (reduce, broadcast, barrier), the operation completes when the last node finishes. If 199 nodes complete in 1ms and one node takes 5ms, the collective takes 5ms. Every node waits for the straggler.

This is tail latency. Not average latency — the worst case that everyone pays for.

The math was unforgiving:

• 200 nodes meant 200 chances for something to be slow
• Network jitter, OS scheduling, memory pressure, thermal throttling — any hiccup on any node became everyone’s hiccup
• Scaling out made this worse, not better

Scaling in — fewer, faster nodes with more cores and memory — reduced synchronization points and kept data closer to compute. The “cluster” that worked best was the smallest one that fit the workload.

Twenty Years Later, Same Physics

The numbers changed. The physics didn’t.

In 2006, my Beowulf cluster taught me that interconnect latency dominates distributed compute. In 2015, Chad’s hierarchy gave me the framework. In 2026, designing InfiniBand topologies, the same lesson keeps emerging:

Scale in first. Scale out when you must.

NVLink in the hundreds-of-nanoseconds range beats InfiniBand in the single-digit-microseconds range. For tightly-coupled workloads running thousands of collectives per training step, that gap compounds fast. The NVLink domain often outperforms the equivalent InfiniBand cluster not because of raw bandwidth, but because of what’s not in the path: no PCIe traversal to a NIC, no driver stack, no serialization across a fabric, no reassembly on the remote side.

Fewer hops. Tighter integration. Lower tail latency.

The game hasn’t changed.

What a Nanosecond Actually Means

Here’s the problem with discussing nanoseconds: humans can’t feel them.

An eye blink takes about 150 milliseconds. That’s 150,000,000 nanoseconds. We have no biological reference for timescales a hundred million times faster than our reflexes.

So let’s slow it down.

Scale: 1 nanosecond = 1 second of human time.

At this scale, an eye blink takes about 4.75 years. A one-second pause in conversation becomes 31.7 years — an entire generation.

Now watch what happens when a CPU sends a write to storage — each row represents a single operation or hop, not a full transaction:

Think about that.

The CPU asks for data from L1 cache and gets an answer in one second of human time. It asks for the same data from a SAN, and waits twenty-three days.

A local NVMe read returns in under 3 hours. The SAN takes more than three weeks.

This is why the storage industry spent decades moving from SAN to local NVMe. The latency gap isn’t incremental — it’s the difference between a quick question and a month-long expedition.

Now apply the same scale to GPU interconnect — again, these are single-hop or small-message latencies, not full collective operations:

A single small exchange via NVLink is a quick hallway conversation. The same exchange across InfiniBand is a scheduled meeting with travel time.

Now consider that an all-reduce collective involves many such exchanges, layered with synchronization and reduction overhead. Multiply those human-scale times by the number of steps in the algorithm, and the accumulated difference between staying on-package versus crossing a fabric becomes weeks versus months.

This is why NVIDIA builds racks of 72 GPUs connected by two miles of copper cable instead of just plugging them into a network switch.

The win isn’t that copper is inherently faster than fiber — it’s that keeping everything in a tightly-integrated domain eliminates the layers that add latency: no NIC, no host PCIe traversal, no driver stack, no switch pipeline per hop. At human scale, NVLink is a conversation across a conference table. InfiniBand is a phone call routed through a switchboard. PCIe is sending a letter.

When your training run makes a trillion of these exchanges, the path matters more than the payload.

Tail latency commands the game because the slowest path commands the game. And physics doesn’t negotiate.

What’s Next

This is the first in a series.

The latency hierarchy doesn’t just govern GPU interconnect — it governs every distributed system decision that matters. And in large teams, these details get buried. Someone writes an async flush assuming it’s fire-and-forget. Someone else configures geo-redundant writes without understanding the round-trip cost. The application works in dev, passes staging, and falls apart at scale.

The physics doesn’t change because you stopped paying attention.

Upcoming articles will dig into the patterns that break silently:

• Sync vs. async writes in geo-redundant clusters — when “durable” means waiting for a packet to cross an ocean
• Consensus protocols and the speed of light — why your distributed database can’t outrun physics
• Storage tiering under pressure — what happens when the hot tier fills and your P99 goes to disk
• The GC pause that killed the training run — how milliseconds of JVM overhead become hours of wasted GPU time

The goal is simple: make the invisible visible. If you can’t see the latency, you can’t fix it.

And if you’re not looking, your competitors are.