Solutions

Multi-plane ZCube @ 100,000 GPUs

Multi-port NICs (4 ports or more) can form a multi-plane ZCube architecture and raise scalability.

[Multi-port NICs, 100,000 GPUs]

Intended audience

This solution is designed for:

Multi-port NICs, or NICs that support port split, such as 4x200G CX8 and CX9

Maximum scale

Maximum ZCube interconnect scale depends on NIC bandwidth and switch specification.

4x200G NICs, 102.4T switches (256x400G)

Maximum ZCube interconnect scale: 65,536 NICs

8x100G NICs, 102.4T switches (256x400G)

Maximum ZCube interconnect scale: 262,144 NICs

Solution details

ZCube networking designs at each scale.

Two-plane ZCube @ 65,536 GPUs

Example with 102.4T switches

Two-plane ZCube 65,536-GPU networking topology

Four-plane ZCube @ 131,072 GPUs

Example with 102.4T switches

Four-plane ZCube 131,072-GPU networking topology

Expected value

At every scale, ZCube uses fewer network devices than Clos.

Lower network hardware cost

33% fewer network switches

33% fewer 400G modules

25% fewer fiber cables

Construction shortened to 5–7 days

Solutions

Other solutions

See more scalable ZCube networking designs by cluster scale, NIC form, and switch specification.

[Dual-port NICs, 16,384 GPUs]

Dual-port ZCube @ 16,384 GPUs

A scalable ZCube architecture designed for dual-port NICs such as CX7 and CX8.

Intended audience: Dual-port NICs, or single-port NICs that support port split
Maximum scale: 51.2T switches can interconnect 16,384 GPUs
Solution details: See details
Expected value: Fewer network devices

View solution details

[Single-port NICs, 8,192 GPUs]

Single-port ZCube @ 8,192 GPUs

A scalable ZCube architecture designed for single-port NICs that can form an equivalent dual-NIC pair.

Intended audience: PCIe topology that meets affinity requirements and can form an equivalent dual-NIC pair
Maximum scale: 51.2T switches can interconnect 8,192 GPUs
Solution details: See details
Expected value: Fewer network devices and stronger fault tolerance

View solution details

[B300, 16,384 GPUs]

B300 ZCube @ 16,384 GPUs

A scalable ZCube architecture designed for B300 clusters.

Intended audience: B300 GPUs with 2x400G NICs
Maximum scale: 102.4T switches can interconnect 16,384 GPUs
Solution details: See details
Expected value: Fewer network devices

View solution details

Start with the network

Make the next AI data center more efficient, starting with the network

Whether you are building a new cluster, upgrading an existing network, tuning performance, or deploying intelligent operations, we can assess how the network can unlock more effective compute.