1. The Claim
If you had asked in 2024 how much research space remained in collective communication, the common answer was: NCCL is mature, Ring/Tree algorithms are near their theoretical bounds, what's left is engineering tuning. SIGCOMM 2026 answered with three full sessions (RS3 Collective Communication Algorithms, five papers; RS5 In-Network Aggregation for ML, five; RS11 Scheduling for ML Training Clusters, five) plus one from ES1 — sixteen slots in all (Turbo was close-read in Piece 1; this piece covers the other fifteen, and only Balancing from ES1 counts here — the other three belong to the AI-datacenter-network thread). The space is not just open; it is opening a new era: collective communication libraries are evolving from "a library with a set of predefined algorithms" into "runtime systems with telemetry, decision-making, and hot-swapping."
The driving forces are not academic inventions but three production realities:
One: MoE explodes communication patterns. Dense-model DP training needs only AllReduce — regular and predictable. MoE's expert parallelism (EP) introduces All-to-All: tokens are routed dynamically by a learned gating function to experts scattered across GPUs, so volume, destinations, and skew all drift with training. Theseus's motivating measurements show expert-load skew drifting over hours; schedules optimized for balance decay into suboptimality. UBEP lists CANN EP's BSP execution model as its renovation target. Alibaba's Balancing and Beyond confirms from production (Qwen3-Coder, DeepSeek-R1) that EP stragglers must be solved at the communication layer, not pushed onto load balancing.
Two: Superpods and dual-plane networks make topologies heterogeneous. NVL72/576 and Huawei CloudMatrix384 integrate hundreds of dies into one logical domain at extreme bandwidth — but the interior is multi-tier (UB's base/opt layers), with non-uniform physical latency. UBEP's three motivating bottlenecks — BSP serialization, the synchronization tax, topology-blind scheduling — all stem from "a unified logical address space ≠ physical uniformity." And beyond the superpod lies the DCN plane: collectives now split work across two planes (NCCL's multi-channel mechanism across NVLink/IB is the primitive form).
Three: Multi-tenancy makes fairness first-class. LEVELLER observes that when ten LLMs (GLM/Gemma/DeepSeek families) share a cluster, per-flow fairness fails under synchronization semantics — a slow job's gradient sync blocks the fast ones. MonkeyTree defragments via job migration; PReCCL reallocates work across virtual topologies under contention. Fairness, fault tolerance, workload redistribution — the classic operating-systems agenda is entering the communication library, word for word.
2. What the Papers Actually Do
2.1 Theseus: runtime admission and hot-swapping of schedules
The NCCL era left a hidden legacy: the schedule space is fixed at initialization. NCCL ships 13 predefined schedules and an alpha-beta cost model that picks one at launch, never to change. Once training runs, conditions drift: MoE expert-load skew moves over time; links degrade, nodes fail. A schedule optimized for launch-time conditions becomes, hours later, suboptimal or a failure point. Theseus (PKU's Qun Huang group + Alibaba Cloud) attacks exactly this "obsolete at initialization" structure.
It gives the communication library two runtime mechanisms. Dynamic selection: users define selection contexts over cluster-wide runtime attributes; a customized agreement protocol keeps every GPU choosing consistently, at near-zero amortized overhead. Hot swapping: schedule resources are organized in a tree and shared across schedules; a swap performs delta migration, building only the increment rather than from scratch.
The fail-slow scenario shows what the machinery is worth: a NIC rate-limited to 10% degrades NCCL's iteration time from 1.75s to 3.76s; at 0% (a dead link), NCCL training halts outright, while Theseus's schedule adaptation restores 2.03s — only 1.16x of the normal iteration. The full ledger: up to 1.61x stable-state and 2.46x dynamic-environment communication speedup (vs NCCL), fail-slow end-to-end JCT (job completion time) 1.84x; MoE EP32 training efficiency 1.07x — 8 wall-clock hours saved over 100k iterations.
Reading the 1.61x requires knowing the comparison mixes framework and space contributions (the MSCCL++-family schedule space is itself more advanced, as the authors note); the numbers are modest, but they are squeezed from an already highly optimized system. The principle is clear: the schedule must become a runtime-replaceable module, not an initialization-time constant.
2.2 OptCCL: the theory clears the field
Algorithm synthesizers were long stuck in a pick-two trilemma: optimality, scalability, generality — choose two. TE-CCL/SCCL needed three hours at 16-32 GPUs; the general ones were low-quality; the good ones were narrow. Cornell's Shapley/Agarwal/Shmoys (Shmoys a theorist of scheduling and approximation algorithms) dismantled the trilemma with two results: a space-time decoupling theorem proving path selection and per-link scheduling can be separated while preserving optimality, and symmetry contraction proving an optimal solution exists within the hardware-symmetric subspace. The search space compresses mathematically; "cannot coexist" becomes "can coexist". Multiple concurrent collectives become a first-class synthesis input for the first time.
Optimal algorithms for hundreds of GPUs in tens of minutes (against TE-CCL/SCCL's three hours at 16-32 GPUs); algorithm quality up to 18x versus TACCL (A100/DOE topologies).
A theory paper in a systems venue sets anchors: it proves optimal + scalable + general can coexist, pulling the heuristic arms race back into an optimization framework. Its relation to Theseus is a relay: OptCCL synthesizes optimal schedules offline; Theseus admits and hot-swaps them at runtime. One link in the relay is still missing (see §3, Observation 1).
2.3 UBEP: re-architecting the EP library for superpods
The superpod ledger carries three hidden taxes. Hundreds of dies interconnect into one logical domain, but the interior is multi-tier (base/opt layers) with non-uniform physical latency; and CANN EP's BSP (bulk-synchronous) execution model absorbs all of it: a serialization tax (independent communication phases cannot overlap), an explicit-sync tax (flags and barriers on separate paths, fully exposed on low-latency fabrics), and a topology-blind tax (scheduling by token count, blind to physical latency). After link bandwidth scales exponentially, microsecond fixed costs dominate — when transfer approaches free, the protocol tax is the new bottleneck. Structurally the same motivation as kernel bypass in RDMA history, at a finer scale.
NJU + Huawei (11 of the authors) rebuild the EP library with three mechanisms, one per tax: dependency-driven execution decomposes All-to-All into fine-grained tasks scheduled on data availability, eliminating global barriers; Data-as-Flag embeds sync signals in data payloads via atomic instructions, near-zero-overhead implicit synchronization replacing explicit flags; topology-aware scheduling puts the multi-tier fabric's non-uniform latency into the decision variables.
Validated on up to 256 NPU dies allocated from a production CloudMatrix384 (the paper's own wording): All-to-All latency down up to 52.4%, end-to-end TPOT (time per output token) down 11.1%.
Effective, but the baseline needs stating first: the comparison is same-hardware CANN EP. NVLink DeepEP cannot run on a CM384; a cross-ecosystem baseline does not exist by construction — an inherent boundary of domestic-supernode papers, not a flaw.
2.4 EPIC: standardizing in-network collectives for open Ethernet
NVIDIA SHARP's in-network aggregation is InfiniBand's proprietary weapon; the Ethernet world has no equivalent. That is not just a technical gap but a structural hole in the industry: every time the open ecosystem wants INC, it waits for a chip or system vendor to ship a private protocol.
PKU (Wenfei Wu, corresponding) + Centec + Lenovo + Alibaba + Infrawaves + NUDT, thirty-plus authors — a working-group paper. Twenty-eight pages do one thing: define an INC protocol specification for Ethernet. A unified abstraction (participant roles, functional boundaries, interoperable interfaces) as the base; polymorphic realization layered by hardware capability, so vendors can iterate incrementally from the simplest polymorphic mode to full aggregation; every mode formally verified; validation across five hardware classes from Tofino/NP testbeds through FPGA/RTL.
The engineering is solid, but the industrial signal outweighs the technical one: a Chinese industry alliance — domestic switch-silicon vendor Centec, server vendor Lenovo, hyperscaler Alibaba, plus universities — acting jointly on Ethernet INC standardization, appearing as a SIGCOMM main-track paper for the first time. Its fate rests on adoption, not on the paper; but appearing in the main track is itself the starting signal of an ecosystem offensive.
2.5 PReCCL: telemetry built into the communication library
NCCL decomposes collectives across virtual topologies (VTs) with static equal work-splitting. Under multi-tenant contention, the slowest VT gates completion while healthy VTs idle — a resource mismatch with no in-library correction mechanism.
Tsinghua + ZGC Lab + Alibaba Cloud build software inband telemetry into the CCL: per-VT stall counts, with telemetry metadata piggybacked on existing collective traffic — no P4 switches required — plus an epoch-based deterministic protocol reallocating work across VTs at collective boundaries. CCT up to 2.1x, end-to-end training 1.21x.
Beyond the numbers, what matters is the validation scale: a 1024-GPU production cluster — the only production-scale thousand-GPU data point among the fifteen papers covered here. Deployability is also the highest tier: no new hardware, NCCL-patch level.
2.6 DynamiQ: quantization must be designed for the topology
Gradient-compression schemes mostly assume a single-hop parameter server: quantize, send, aggregate, done. In multi-hop aggregation (ring/butterfly), quantization error on partial sums accumulates hop by hop; the bandwidth saved can be cancelled by the compounding precision loss.
UCL + Broadcom + Harvard (Mitzenmacher) designs quantization for the topology: two-phase quantization allocates bits by coordinate magnitude, and a fused decompress-accumulate-recompress kernel rebalances error mid-flight so it cannot snowball. Across all evaluations, it is the only method keeping 99.9% of BF16 baseline accuracy, while gaining up to 34.2% over the strongest baselines (OmniReduce, THC, MXFP4/6/8).
Effectiveness is solid, and Broadcom's authorship is the more telling industrial signal: a network-silicon vendor working directly at the communication library's algorithm layer. Read together with Centec in EPIC and Huawei in UBEP, the switch-silicon camp is placing pieces across the communication software stack.
3. Route Analysis: Five Routes on One Map
Fifteen papers, five routes:
Three structural observations:
Observation 1: Offline synthesis and runtime adaptation are forming a relay. OptCCL solves "what is optimal" (offline, theoretical); Theseus/PReCCL solve "how to stay optimal while drifting" (online, systemic). Today's gap is in between: runtime-triggered incremental re-synthesis. Theseus already has the telemetry-decision-hotswap loop; its schedule space remains predefined, and OptCCL's synthesis stays offline — nobody has connected the two. Whoever does takes the next-generation NCCL position.
Observation 2: Compression sinks from the application layer into the communication library. ZipCCL (lossless, exploiting the near-Gaussian distribution of LLM tensors — communication time down up to 1.35×, end-to-end 1.18×, zero quality impact) and DynamiQ appearing in the same conference is no coincidence: once the library becomes a runtime, compression is its data plane. This replays video-coding history — codecs moving from application code into the transport stack.
Observation 3: Three hardware routes converge on in-network aggregation. Turbo (programmable-switch lookup tables), EPIC (Ethernet protocol specification), HyNA (custom aggregation on switch silicon — 7.35× over BytePS with <3% area, on a 100Gbps FPGA prototype plus 7nm ASIC synthesis) — one goal, three substrates. UBEP moves the other way (synchronization into data payloads, no network support needed). The cost/flexibility triangle among RMT programmability, ASIC customization, and protocol standardization will resolve within two years.
4. Evidence Check
Headline numbers anchored (per-paper details live in the fact cards):
| Claim | Number | Source |
|---|---|---|
| NCCL ships 13 predefined schedules | "13 predefined schedules" (PDF) | Theseus §2.2 |
| Theseus 1.61× stable / 2.46× dynamic / 1.84× fail-slow JCT | vs NCCL | Theseus abstract |
| OptCCL: hundreds of GPUs in tens of minutes | vs TE-CCL/SCCL 3h@16–32 GPUs | OptCCL abstract/§7.1 |
| OptCCL quality up to 18× vs TACCL | A100/DOE topology experiments | OptCCL §7.1 |
| UBEP −52.4% All-to-All / −11.1% TPOT | 256 dies allocated from production CM384, vs CANN EP | UBEP abstract |
| EPIC: 28 pages, five validation classes | Tofino/NP/FPGA/RTL | EPIC structure |
| PReCCL CCT 2.1× / training 1.21× | 32-GPU testbed + 1024-GPU production cluster | PReCCL abstract/§6 |
| DynamiQ 99.9% accuracy + 34.2% speedup | vs OmniReduce/THC/MXFP4 | DynamiQ abstract |
| ZipCCL 1.35× comm / 1.18× e2e, lossless | 64-GPU cluster | ZipCCL abstract |
| HyNA 7.35× over BytePS, <3% silicon area | 100Gbps FPGA prototype + 7nm ASIC synthesis | HyNA abstract |
| LEVELLER min-rate improvement 28%–120× | 10 co-trained LLMs | LEVELLER abstract |
Provenance notes: UBEP's baseline is CANN EP — a same-ecosystem engineering adaptation (a NVLink-DeepEP comparison on CM384 is impossible by construction; this is an inherent boundary of domestic-supernode papers, not a flaw). PReCCL's 1024-GPU number is a production deployment measurement at Alibaba.
5. Assessment and Predictions
1. NCCL's moat will be eroded from both ends, but not breach immediately. Library compatibility (PyTorch integration) plus the CUDA ecosystem remains a real barrier — Theseus and PReCCL both target NCCL drop-in replacement, which proves the point. But MoE/EP is a workload NCCL was never designed for: UBEP already bypasses NCCL inside the Huawei ecosystem, and UCCL-class newcomers fill in elsewhere. Within two to three years, the communication-library market becomes load-aware multi-library coexistence, with a Theseus-class runtime as the scheduling entry point.
2. EP communication is the critical battleground for domestic supernode stacks. UBEP proves Huawei has taken its EP library to production grade inside the UB ecosystem; the EPIC working group attempts to give China's industry alliance a leading hand in Ethernet INC standardization. This cashes the judgment from our UnifiedBus analysis — the interconnect protocol is only the foundation; the communication library above it is the user-perceivable performance surface.
3. Fairness scheduling erupts in 2027. This year's five papers (MonkeyTree/LEVELLER/Aegis/GeoOrchestra/Disaggregated RL) are scattered probes; as inference and training clusters converge (PD disaggregation + elastic scheduling), communication fairness becomes the first constraint on utilization. China Telecom Cloud's LEVELLER shows operator-backed AI clouds already hit this in real multi-tenancy.
4. Deployability in three tiers. Immediately usable: PReCCL's software telemetry (no new hardware; NCCL-patch level), ZipCCL/DynamiQ compression kernels (library-level insertion). Mid-term (with hardware refreshes): EPIC's polymorphic INC (Centec/Lenovo silicon and systems), HyNA's ASIC aggregation (next switch-chip tapeout window). Ecosystem-locked: UBEP (UB atomic semantics — unportable to NVLink; both Huawei's moat and UBEP's ceiling).
5. A contrarian read: the density of RS3/RS5 itself signals that both lines are near their submission peak. OptCCL's theoretical field-clearing usually marks a direction's maturity. The next window is the runtime layer — the full loop already exists in Theseus; the gap is runtime-triggered incremental re-synthesis, where the schedule space itself must be regenerable online.
Disclosure: Based on fifteen papers: six read in full (Theseus/OptCCL/UBEP/EPIC/PReCCL/DynamiQ), four at medium depth, five surveyed (abstract-level, cited qualitatively only). UBEP's baseline is same-hardware CANN EP — an intra-ecosystem comparison; no cross-ecosystem baseline exists by construction. Data as of August 22, 2026.
Piece 2 of the SIGCOMM 2026 close-reading series. Previous: the overview (Piece 0), KV cache as network citizen (Piece 1). Next: "Scale-Up Is the New Datacenter Network."