ZCube: Mitigating AI-Traffic Path Conflicts with Structured Network Load Balancing

01How Does Data Travel When a Pair of GPUs Communicate?

Assume GPU-A and GPU-B are not connected beneath the same switch.

GPU-A has two network entry points: an odd-group switch OA and an even-group switch EA.

GPU-B also has two entry points: an odd-group switch OB and an even-group switch EB.

Because the odd and even switch groups are fully interconnected, the following candidate path exists:

GPU-A → OA → EB → GPU-B

As does:

GPU-A → EA → OB → GPU-B

In the physical topology, both directions have two-switch paths. ZCube's routing policy selects a unique optimal path for every GPU pair.

The key is not to let switches hash arbitrarily across multiple equal-cost paths. Instead, it: assigns path ownership to each GPU pair in advance according to the topology.

Official material states that ZCube routing can determine a unique optimal path for each GPU pair, avoiding traffic conflicts caused by multipath selection.

Accordingly, official material describes ZCube's network diameter as two switches for communication between any GPU pair.

A cross-Leaf path in a conventional two-tier Clos fabric usually traverses three switches: the source Leaf, Spine, and destination Leaf.

ZCube does not “find many paths and then hash”; it uses dual access and a bipartite graph to assign deterministic, short paths to GPU pairs.

02Why Can ZCube Be More Balanced without Relying on ECMP?

The conventional Clos approach is: provide multiple equal-cost paths physically, then distribute traffic with ECMP hashing. The problem is that a hash does not know: how large a flow is, whether more flows of the same kind will appear next, whether a link is already carrying a large KV flow, or whether two source-destination pairs will collide. For many short flows, hashing is usually good enough.

But AI clusters commonly have: not necessarily many flows, but very large flows; synchronized traffic; and highly concentrated communication relationships. That is when hash collisions occur.

ZCube takes a different approach: it first maps GPUs to two complementary switch groupings through the topology itself, then uses the complete bipartite graph and dedicated routing to assign a unique optimal path to each GPU pair. In other words, it addresses at the architectural level: which flows should not collide in the first place.

The official ZCube report divides congestion into two categories: unavoidable congestion, created when multiple sources send to the same destination simultaneously; and avoidable congestion, caused by topology, traffic mapping, and uneven multipath utilization.

ZCube primarily targets the second category.

ECMP means “after a collision occurs, hope the distribution is sufficiently even in probability”; ZCube means “during topology and path design, avoid collisions that should not happen.”

This is the fundamental difference between the two approaches.

03Why Does This Structure Suit Both AllReduce and All-to-All?

An AI network carries more than one kind of traffic.

AllReduce is typically characterized by: multiple ranks participating in a reduction, with data passed in stages through a ring, tree, or another algorithm. Communication relationships are regular. All-to-All is different: each rank sends different data to every other rank. In MoE scenarios, it can create more complex many-to-many traffic.

In NCCL's official definitions, AllReduce gives every rank the reduced result; All-to-All requires every rank to send distinct data blocks to all other ranks and receive data from every rank.

ZCube's two GPU-access modes provide two complementary perspectives. Contiguous grouping (not rail-optimized) preserves local relationships. Cross grouping (rail-optimized) distributes GPUs with the same relative index across different switches.

Combined with the complete bipartite interconnection between the two switch groups, this reduces the probability that different communication pairs concentrate on a small number of intermediate nodes.

Official material states that ZCube can achieve better fabric-wide load balancing for typical training traffic such as AllReduce and All-to-All, as well as inference traffic with uncertain source-destination relationships and uneven NIC loads.

But note: ZCube does not perform AllReduce. Nor does it perform reductions on NCCL's behalf.

ZCube is responsible for: placing these communication relationships on more appropriate physical paths.

NCCL decides “who exchanges what with whom”; ZCube decides “where those communications should travel whenever possible.”

4Why Is ZCube Particularly Attractive for P/D-Disaggregated Inference?

After P/D disaggregation, the network gains a critically important traffic type: KV Cache Transfer

Prefill generates the KV Cache. Decode needs to read it. If P and D are not on the same server, it must move across the network. The issue is that KV traffic is not as regular as training AllReduce.

Requests differ in prompt length, KV size, selected P node, selected D node, cache-hit location, and continually changing scheduling relationships. This creates pronounced asymmetry.

For example:

P1 → D3: 200 MB
P2 → D3: 300 MB
P4 → D3: 500 MB
P5 → D7: 50 MB

A Decode node, NIC, or Leaf switch can suddenly become a hotspot.

In a production report released by ZCube, the original ROFT network had persistently hot Leaf switches, egress queues that remained deeply occupied, and frequent PFC backpressure. The report attributed the root cause to a mismatch between P/D traffic and conventional rail mapping.

ZCube does not aim to make all KV traffic disappear. It distributes different P-D pairs across a wider space of inter-group links. This can reduce cases where: multiple independent large KV flows are forced onto the same Leaf or Spine path because of topology mapping.

ZCube does not optimize the KV Cache itself; it optimizes how KV Caches “meet” within the cluster.

5Can ZCube Solve Incast? It Must Not Be Overstated

ZCube can reduce avoidable congestion caused by topology. But it cannot eliminate all congestion.

Assume 20 Prefill nodes send KV Caches to the same Decode node simultaneously. No matter how many intermediate paths exist, they must ultimately converge on: the same destination NIC, server downlink, and switch egress queue. That is Incast. Intermediate paths can be distributed. The final hop cannot become wider by itself.

ZCube also makes the distinction clear: congestion created by multiple GPUs sending to the same destination simultaneously is unavoidable and still requires mechanisms such as congestion control and traffic shaping.

RoCE documentation likewise notes that RoCE networks still rely on mechanisms such as ECN and PFC.

ZCube can prevent everyone from incorrectly crowding onto one path, but not everyone from needing to reach the same destination.

This is the most important statement for understanding its limits.

In summary, ZCube replaces the passive traffic distribution of conventional Clos networks, which relies on random ECMP hashing, with a bipartite topology and deterministic short-path routing. It avoids avoidable congestion in AI training and P/D-disaggregated inference at the architectural level and balances traffic across the fabric. It cannot, however, eradicate destination-convergent Incast congestion and must work with RoCE congestion control and NCCL communication algorithms.