Routing and Congestion in 100K-GPU Clusters: When Physics Pushes Back
A 100K-GPU cluster network isn't just a bigger network — it's a network governed by different physics. This article systematically breaks down eight core challenges in routing and congestion control, and examines how the industry's four approaches (RoCEv2, MRC, UET, InfiniBand) are responding — and where the gaps remain.
This article continues the locsic.com AI networking series. Previous installments: From CLOS to ZCube: Network Topology Evolution in Intelligent Computing Clusters, MRC Protocol: When the NIC Becomes the Brain, Scale-Across: AI Clusters Are Growing Across Cities.
The Physics of the Problem
A 100K-GPU cluster network faces not an abstract problem, but three physical quantities degrading simultaneously:
| Physical Quantity | ~10K GPUs | ~100K GPUs | Consequence |
|---|---|---|---|
| Concurrent links | ~tens of thousands | ~hundreds of thousands | ECMP hash collisions go from probabilistic to inevitable |
| GPUs participating in a single comm step | ~8,000 | ~100,000 | AllReduce/All-to-All incast scale grows 12× |
| Packets in flight at any moment | ~millions | ~tens of millions | Any minor congestion amplifies into system-wide buffer pressure |
The compounding effect: problems solvable by tuning at 10K GPUs become structural bottlenecks at 100K. Synchronous pretraining's traffic pattern makes this bottleneck more lethal. Each training step is lockstep — compute in parallel, then communicate globally. The slowest single transfer in a communication round determines the entire step's duration. Tail latency dominates performance, and tail latency is the direct product of routing and congestion control failure.
Routing Challenges
Challenge 1: ECMP Hash Collisions — Uneven Traffic Distribution
Traditional RoCEv2 networks use ECMP (Equal-Cost Multi-Path) for load balancing: switches hash the five-tuple of each packet and select an equal-cost path. This is flow-level assignment — all packets in the same flow take the same path.
In a 100K-GPU cluster, millions of micro-flows exist simultaneously. Hashing is probabilistic. When link counts reach tens of thousands and flow counts reach millions, multiple large flows inevitably hash onto the same link. A single link selected by two AllReduce elephants simultaneously causes latency to spike, dragging the entire step's tail latency.
The cruelty of scale manifests as a birthday paradox effect. Assume the probability of any single link being selected is 1/N (N equal-cost paths). At 10K GPUs, N might be 64 — collision probability is acceptable. At 100K GPUs, even if topology growth pushes N to 256, the 10× increase in flow count means the probability of at least one large-flow collision still approaches 1.
Industry approaches:
| Approach | Core Mechanism | Representative |
|---|---|---|
| Packet spraying | Breaks flow-level binding, distributes packets from the same flow across 128–256 paths | MRC (EV spraying), UET (packet spraying) |
| Adaptive routing | Switches dynamically select paths based on port utilization | NVIDIA Spectrum-X Adaptive Routing, IB Adaptive Routing |
| Topology-aware routing | Pre-computes optimal paths using topology structure to reduce collision probability | ZCube/ATOP topology search, Google Jupiter egress routing |
| Source routing | Sender encodes the full path in the packet header, precisely controlling each hop | MRC + SRv6 uSID |
The core divide: packet spraying eliminates hash collisions entirely but requires the receiver to handle out-of-order delivery (NICs need an OOO write engine). Adaptive routing preserves in-order delivery but requires switches to maintain port utilization state, and decision quality is limited by the switch's local view. Source routing gives full control to the NIC but requires global topology knowledge and a path orchestration tool.
Topology choice is also reshaping the routing problem. Traditional three-tier Fat-Tree ECMP inherently carries hash collision risk, while Dragonfly topologies (deployed by HPE Slingshot at Frontier/El Capitan) use adaptive routing to bypass ECMP, but introduce unpredictable behavior under congestion. ZCube's asymmetric topology breaks the assumption that all switch ports have equal count, and actually pairs better with source routing: source routing doesn't require paths to be "equal-cost" — the NIC can schedule precisely based on each path's actual state. MRC's 8-plane two-tier Clos compresses the path within each plane to 3 hops, shortening the reordering window for packet spraying. Topology, routing protocol, and congestion strategy are a co-design problem — they cannot be discussed in isolation.
Challenge 2: Dynamic Routing Convergence Latency — Recovery Too Slow
Link and switch failures in a 100K-GPU cluster are not a question of "if" but "how many times per day." BGP/OSPF/IS-IS must detect the failure, propagate the routing update, recompute shortest paths, and update the FIB. This entire convergence process in large clusters can take tens to hundreds of milliseconds.
Training jobs are extremely latency-sensitive. During a routing convergence event, affected AllReduce operations will time out. The consequence is not slowdown — it's the entire training job restarting from the last checkpoint, potentially losing tens of minutes to hours of GPU time.
Worse, post-convergence traffic redistribution can trigger new ECMP hash imbalance, causing secondary congestion.
Industry approaches:
| Approach | Recovery Speed | Mechanism |
|---|---|---|
| MRC + SRv6 static source routing | Microsecond-level | NIC detects path failure → retires the corresponding EV → switches to a new path; switches are not involved |
| UET adaptive routing | Millisecond-level | Switches adjust routing based on ECN feedback; NIC doesn't need topology awareness |
| InfiniBand adaptive routing | Second-level | SM builds base routing table (topology-level); hardware does packet-level adaptive scheduling within the table |
| Traditional BGP convergence | Seconds to tens of seconds | Detect failure → update propagation → SPF computation → FIB update |
MRC made a key design decision: completely disable switch-level dynamic routing, concentrating all routing intelligence in the NIC. The paper's rationale: two adaptive systems (NIC doing path selection + switch doing route convergence) operating simultaneously interfere with each other. The NIC knows which EVs are healthy and which are congested, enabling precise decisions; the switch has only a local view and may make decisions that conflict with the NIC.
Challenge 3: Cross-Domain Routing (Scale-Across)
When clusters expand from a single datacenter to metro or inter-city deployments (Meta, Microsoft, and ByteDance are all doing this), routing faces a new dimension: fiber latency. Metro RTT is ~1ms; inter-city RTT is 10–50ms. If routing protocols are latency-oblivious and mix high-latency and low-latency paths into ECMP, the result is massive tail latency disparity.
Ring AllReduce is extremely sensitive to path latency consistency — the slowest hop in the ring determines overall speed. Cross-domain deployment requires routing protocols to distinguish "near paths" from "far paths" and never treat paths with different latencies as equal-cost.
Multiple approaches are being explored in parallel. Upper-tier switches can perform latency-aware routing, distinguishing near and far paths. On the training framework side, Megatron/DeepSpeed parallelism planners consider network topology when assigning GPUs to physical locations, prioritizing communication-intensive parallelism groups within the same datacenter. NCCL automatically detects NUMA/NIC/switch affinity and prefers near paths when building communication rings. At the algorithm level, switching cross-domain communication from synchronous AllReduce to pipelined asynchronous communication (pipeline parallelism naturally tolerates higher latency) reduces dependence on low-latency routing. For dedicated DCI, Google's B4/Jupiter and Microsoft's ExpressRoute use SDN-controlled cross-domain routing.
There is currently no mature general-purpose solution. The most practical path is framework-side adaptation (NCCL topology awareness + pipeline parallelism across domains). Network-layer latency-aware routing is still in the custom-build phase. Meta's two 24K H100 clusters span multiple datacenters, with their RouteInfra system doing extensive custom adaptation; ByteDance's ByteScale framework optimizes communication scheduling at the framework layer rather than relying solely on the network layer. Most hyperscale users build custom adaptations.
Challenge 4: Management Complexity of Path Orchestration
Traditional network routing is "self-healing": switches run routing protocols automatically, and administrators don't need to care which path each flow takes. But MRC's SRv6 source routing and ZCube's topology search both require centralized path orchestration. A controller must maintain global topology state, compute optimal paths for each GPU pair, assign uSID encodings, and recompute on failure.
OpenAI has not published implementation details for this component. But from an engineering perspective, we can infer:
- Real-time maintenance of 131K-GPU global topology state
- Post-failure path recompletion within milliseconds
- uSID assignment collision avoidance
- Incremental updates triggered by switch add/remove events
This is a domain that has not yet been productized. There is no commercial SRv6 path orchestrator on the market that supports 100K-GPU scale.
Congestion Control Challenges
Challenge 5: PFC Storms — Cascading Failures in Lossless Networks
RoCEv2 relies on PFC (Priority Flow Control) to achieve losslessness: when the receiver's buffer nears capacity, it sends a pause frame to make the upstream switch stop transmitting. Pause frames propagate upstream, creating head-of-line blocking.
In a 100K-GPU cluster's three-tier Clos, a single pause frame's propagation chain can span 5–7 switch hops. Any incast point (multiple senders simultaneously transmitting to one receiver) can trigger a pause storm, expanding the impact from a single link to an entire pod. More severe is PFC deadlock: pause frames of different priorities blocking each other in a circular dependency. Once deadlock occurs, the entire communication path permanently stalls, requiring manual intervention. The larger the scale and the more complex the topology, the more deadlock scenarios exist.
Three industry approaches address this problem. MRC disables PFC entirely: switches are allowed to drop packets, and the NIC uses selective ACK for precise retransmission. This aligns with the internet TCP philosophy — reliable transmission over a lossy network. The benefit is complete elimination of PFC storms and deadlock risk; the cost is overhead from packet loss and retransmission. UET and InfiniBand use credit-based flow control: the receiver grants credits to the sender, and packets are sent only when credits are available, preventing buffer overflow at the source. Neither PFC nor packet loss is needed. But the two operate at different levels: IB credits are link-level (per Virtual Lane), while UET credits are end-to-end (NIC to NIC). The cost is that credit management requires hardware support and end-to-end transparency. Traditional RoCEv2 retains PFC, mitigating risk through deadlock detection and buffer overprovisioning — but at 100K-GPU scale, the engineering cost of this path keeps climbing.
Challenge 6: Incast — Instantaneous Many-to-One Congestion
AI training's AllReduce and All-to-All operations naturally produce incast: hundreds or even thousands of GPUs simultaneously sending data to the same GPU (or group of GPUs). The receiving NIC's ingress bandwidth (800G) is instantly saturated, buffer overflows, and PFC or packet loss is triggered.
During the reduce-scatter phase of AllReduce, each GPU receives data fragments from all other GPUs in the ring. A 100K-GPU AllReduce ring may have 256 participants. Even if each flow sends only a 1GB fragment, 256 flows arriving within a short time window aggregate to ~256GB — far exceeding any switch or NIC buffer capacity. The bottleneck here is not the volume of a single flow, but the number of simultaneous arrivals.
Industry approaches:
| Approach | Mechanism |
|---|---|
| Packet trimming | Under congestion, switches strip payload but keep the header forwarding. NIC receives trimmed header → sends NACK for precise retransmission. Distinguishes "congestion loss" from "failure loss" |
| Packet spraying to disperse incast | Scatters packets from the same flow across 256 paths, preventing all traffic from funneling through one switch |
| In-Network Reduction | Switches perform reduce operations (sum/max), merging N incoming flows into 1 outgoing flow. Communication complexity drops from O(N) to O(1) |
| ECN + rate reduction | Switch marks ECN, sender reduces rate upon receipt (DCQCN congestion control) |
| Topology-aware communication scheduling | Training framework arranges communication pairs based on physical topology, reducing cross-switch incast |
In-network reduction is the most thorough solution to incast. Traditional AllReduce has O(N) communication complexity: N GPUs participate in a reduce-scatter + allgather, and each GPU must send and receive N data chunks. NVIDIA SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) performs reduce operations inside the switch — N flows arrive at the switch, which directly performs addition/max, outputting a single result flow. Communication complexity drops from O(N) to O(1), and receiver buffer pressure drops from N× to 1×.
SHARP has evolved to its fourth generation. The Quantum-X800 switch delivers 14.4 TFLOPS of in-network compute with sub-100ns port-to-port latency (announced at GTC 2024), and is widely deployed in large-scale IB clusters.
But the Ethernet ecosystem's in-network compute solutions are not yet mature. The UEC specification defines an INA (In-Network Assistant) interface, and Broadcom Tomahawk 6's programmable pipeline can theoretically perform reduce operations, but commercial products are still distant. If Ethernet switches could natively perform in-network reduce, incast pressure could drop by an order of magnitude. This is a key milestone for Ethernet catching up to IB.
The interplay between collective communication algorithms and routing also deserves attention. Ring AllReduce is extremely sensitive to network latency consistency — the slowest hop in the ring determines overall speed, demanding high routing determinism. Tree AllReduce is more latency-tolerant but suffers worse incast (multi-level tree simultaneous convergence). All-to-All demands the highest bisection bandwidth. Different collective operations suit different routing strategies: Ring pairs well with deterministic source routing (MRC); All-to-All benefits most from packet spraying. Training frameworks (NCCL/CXCCL) should account for underlying routing capabilities when constructing communication plans, but NCCL's adaptation to MRC/UET is still in early stages.
Challenge 7: Congestion Control Tuning — Black Magic at Scale
The standard congestion control for RoCEv2 is DCQCN (Datacenter Quantized Congestion Notification): switches mark ECN → NIC reduces rate upon receipt → gradually recovers. DCQCN has over a dozen parameters (ECN threshold, rate reduction magnitude, recovery speed, timers, etc.), and is extremely difficult to tune properly at large scale.
In a 100K-GPU cluster, traffic patterns differ by location (access vs. aggregation vs. core layer), by training phase (compute vs. communication), and by collective operation type (AllReduce vs. All-to-All vs. AllGather). One parameter set cannot fit all scenarios, but maintaining different configurations for each scenario increases operational complexity.
Consequences of mistuning:
- Too aggressive (ECN threshold too low, rate reduction too severe) → effective bandwidth drops, training slows
- Too conservative (ECN threshold too high, rate reduction insufficient) → buffer overflow, PFC storms / packet loss
Industry approaches:
| Approach | Starting Point |
|---|---|
| DCQCN fine-tuning | Repeatedly test on large-scale clusters to find a "good enough" parameter set. Engineering-feasible but inelegant |
| NSCC (contributed by AMD) | Congestion control based on network-side feedback; incorporated into both UEC and MRC specs |
| MRC's "switch paths, don't slow down" | On receiving ECN, doesn't reduce rate — marks the corresponding EV as congested and switches paths. Redefines congestion control as a routing problem |
| UET congestion control (from Slingshot) | Algorithm tuned from HPE's years of supercomputing experience |
| HPCC (High Precision Congestion Control) | Uses INT (In-Band Network Telemetry) to obtain precise link load for accurate rate reduction. Academic proposal, limited commercial deployment |
MRC's approach differs fundamentally from all traditional approaches. Traditional congestion control reduces rate on ECN — this decreases the sender's output but also lowers overall throughput. MRC doesn't slow down upon receiving an ECN tagged with an EV; instead, it marks that path as congested and switches to another EV to continue full-speed transmission.
Challenge 8: Long Fat Pipe — Cross-Domain Congestion
When clusters span datacenters, fiber latency makes the network pipe "long" and "fat" (BDP = bandwidth × latency increases). Metro cross-datacenter RTT is ~1ms; at 800G link speed, BDP is approximately 100MB — meaning 100MB of packets are always in flight.
Traditional congestion control (DCQCN/TCP) is feedback-based: detect problem → reduce rate → recover. Feedback delay equals RTT. At 1ms RTT, by the time congestion is detected and rate reduction takes effect, 100MB of packets are already in flight — these cannot be recalled and must be absorbed by buffers or dropped.
Inter-city (RTT 10–50ms) is worse: BDP reaches the GB level, and feedback-based congestion control cannot keep up.
UET's credit-based flow control has a natural advantage in cross-domain scenarios: credits are pre-allocated, so the sender knows the receiver's capacity before transmitting — no need to wait for a feedback loop. MRC's packet spraying can distribute traffic across many paths, reducing per-path BDP pressure. A more direct approach is cross-domain bandwidth overprovisioning — reserving headroom on DCI links, trading cost for stability. On the training framework side, switching cross-domain communication from synchronous AllReduce to pipelined asynchronous communication reduces dependence on low latency.
This area is still in early exploration, with no mature general-purpose solution.
Four Approaches Compared
| Dimension | Traditional RoCEv2 | MRC | UET | InfiniBand |
|---|---|---|---|---|
| Routing | ECMP hash (flow-level) | SRv6 source routing + packet spraying (packet-level) | Switch adaptive routing + packet spraying (packet-level) | SM builds table + hardware packet-level adaptive scheduling |
| Failure recovery | Seconds (BGP convergence) | Microseconds (NIC retires EV) | Milliseconds (ECN-driven rerouting) | Seconds (SM recomputes routing table) |
| Flow control | PFC (lossless, storm-prone) | No PFC, allows loss + retransmission | Credit-based (no loss, no PFC) | Credit-based (native lossless) |
| Congestion control | DCQCN (ECN → rate reduction) | ECN → path switch, no rate reduction | UEC CC (Slingshot lineage) | Link-level credit + transport-level CC |
| Incast mitigation | Tune ECN/PFC thresholds | Packet spraying + trimming | Packet spraying + trimming | SHARP in-network reduction |
| Maturity | ★★★★★ | ★★★★ (1 production deployment) | ★★★ (specification stage) | ★★★★★ (NVIDIA full-stack) |
| 100K-GPU ready | ❌ (clear ceiling) | ✅ (131K in production) | ⏳ (12–18 months out) | ⚠️ (economic scale ≤64K GPUs) |
Unresolved Problems
The following problems currently lack good answers:
-
Centralized path orchestration products don't exist. MRC's SRv6 source routing requires a global controller: topology-aware, path-computing, uSID-assigning, failure-recovering. OpenAI built this system in-house but hasn't open-sourced it. There is no commercial SRv6 path orchestrator supporting 100K-GPU scale.
-
Mixed-workload isolation. A 100K-GPU cluster is unlikely to run only one training job. How do communication patterns from different jobs interfere? Current isolation methods are coarse — VLAN/VXLAN segmentation, PFC priority isolation, physical plane separation. Fine-grained, packet-level QoS is nearly impossible at 100K-GPU scale.
-
Standardization of in-network computing. SHARP is mature in the IB ecosystem, but Ethernet's in-network computing is not yet standardized. UEC defines the INA interface but implementation is lagging. If Ethernet switches could natively perform reduce operations, incast pressure could drop by an order of magnitude.
-
Cross-domain congestion control. There is no mature solution for cross-datacenter or inter-city training congestion control. MRC, UET, and IB were all designed for low-latency, single-datacenter networks. Cross-domain scenarios require new congestion control algorithms — possibly combining credit-based and feedback-based approaches, or introducing predictive congestion control.
-
Debugging and observability. When a tail latency problem occurs in a 100K-GPU cluster, how do you pinpoint which link, which switch, which EV is causing the issue? Traditional network diagnostic tools (sFlow, INT, mirroring) have insufficient sampling rates at 800G line speed, and full capture is infeasible. MRC's 256 paths × thousands of QPs = hundreds of thousands of monitoring dimensions — the data volume far exceeds human-readable scale. Observability is the hidden bottleneck for production deployment. NVIDIA's DOCA telemetry framework and SONiC's streaming telemetry have made progress in this direction, but are still far from delivering "an actionable dashboard for operations teams."
Assessment
Routing and Congestion Cannot Be Discussed Separately
The eight challenges in this article may seem numerous, but they stem from a single core contradiction: routing determines traffic distribution; congestion is the consequence of uneven distribution. Traditional networking treats routing and congestion control as two separate problems — switches handle routing (ECMP/BGP), endpoints handle congestion control (DCQCN/PFC). MRC merges them: on receiving ECN, instead of reducing rate, it switches paths, redefining congestion control as a path selection problem. UET uses credit-based flow control to prevent congestion at the source, also bypassing the traditional "congest first, react later" model. The defining characteristic of AI networking in 2026–2028 is the convergence of routing and congestion control.
No Silver Bullets, Only Trade-offs
All four approaches have hard limitations. RoCEv2 has a clear ceiling at 100K-GPU scale, but remains the most cost-effective choice below 30–50K GPUs. MRC has production validation but only from OpenAI, and its path orchestration tools are not productized. UET has the broadest industry consensus but hasn't yet been validated at scale, and software migration costs are high. InfiniBand's full stack is mature but locked to NVIDIA, with an economic scale ceiling around 64K GPUs: beyond that, the 144-port limit of IB switches forces an additional topology layer, driving up both latency and cost. Ethernet switch ASIC total bandwidth already exceeds IB (e.g., Broadcom Tomahawk 5's 51.2T vs Quantum-X800's 115.2T, but Ethernet benefits from more vendor competition), and costs are declining faster. The scale cost advantage comes not just from port count, but from multi-vendor competition and economies of scale.
Choosing a path is fundamentally a bet on which constraint gets lifted first. Betting on MRC means betting that OpenAI's engineering capability is replicable. Betting on UET means betting in the power of industry standards. Betting on IB means betting that NVIDIA's full-stack integration advantage holds.
China's Special Path
Chinese vendors face an additional dimension: NIC chip self-sufficiency. Both MRC and UET depend on US NIC chips (ConnectX-8 / Pollara / Thor Ultra); China currently has no equivalent product. Huawei's Lingqu bypasses NIC hardware limitations from the system layer (kernel + service layer + cloud layer), using software-defined approaches for adaptive routing and congestion control. The performance ceiling falls short of MRC's hardware packet spraying, but under US chip restrictions, it's a realistic path. Domestic switch chips (Centec/Huawei/H3C) lag by one generation in capacity, but MRC's simplification of switch requirements actually lowers the catch-up barrier. China's path will likely involve switches first adopting MRC/UET open protocols, with NICs following through domestic alternatives or partnerships.
Timeline Assessment
- H2 2026: MRC begins second large-scale deployment beyond OpenAI/Microsoft (possibly Google or Meta). UET interoperability testing enters substantive phase.
- 2027: UET 1.0-compatible NICs enter mass production (AMD Pollara 400 leading), first non-OpenAI UET clusters come online. Ethernet in-network computing (INA) prototypes emerge.
- 2028: MRC v2 or UET 2.0 specification released, incorporating the strengths of both. Ethernet's market share at 100K-GPU scale surpasses InfiniBand.
This timeline carries uncertainty. Variables that could accelerate things: NVIDIA opening SHARP to the Ethernet ecosystem, UEC interoperability testing exceeding expectations, China's domestic substitution progressing faster than expected. Variables that could delay things: MRC deployments beyond OpenAI hitting engineering pitfalls, UET software migration costs exceeding expectations, NVIDIA cutting prices to defend IB market share. But the direction is relatively clear: Ethernet replacing IB in large-scale AI training is a high-probability event — the only question is how fast.
Disclaimer: This article is based on the OpenAI MRC paper (arXiv 2605.04333), OCP MRC 1.0 specification, UEC 1.0 specification, RFC 9800 (SRv6 compressed segment list encoding), public materials from NVIDIA/AMD/Broadcom/HPE, and cross-validated with previously published locsic.com analyses including the in-depth MRC protocol analysis, AI networking protocol landscape analysis, and Lingsheng Supercomputing pure-CPU architecture analysis. Not investment advice. Data herein is current as of July 31, 2026.