KV Cache Becomes a Network Citizen: How SIGCOMM 2026 Rewired LLM Inference as a Networking Problem

1. The Claim

In August 2026, ACM SIGCOMM devoted the opening research session of its conference to a single problem: the network transfer of KV cache in disaggregated LLM serving architectures.

This was not routine topic placement. Four of the five papers in Session 1 (LLM Inference & Serving), one in Session 5 (In-Network Aggregation for ML), and one in Session 18 — six papers in total — cover the complete stack from compression policy, transfer encoding, and storage I/O scheduling to endpoint migration and in-switch aggregation. Read together, they reveal one clear picture:

As prefill-decode (PD) disaggregation becomes the default architecture for inference serving, KV cache has turned from GPU-internal state into an explicit payload that crosses the network. In under two years, the networking community has redefined this new traffic class from a performance problem into a research domain of its own.

The timeline is worth recording. PD-disaggregated systems (DistServe, Mooncake, 2024) first deployed in production; the capacity problem (the tyranny of HBM) and the transfer problem first appeared at systems venues (OSDI/SOSP/ATC); by 2026, SIGCOMM has absorbed KV transfer into its core agenda — not as an application backdrop, but as an object of networking research in its own right. Nearly all six papers' introductions recycle the same class of measurement, and both numbers come from KVServe alone: on an H100 decode node swept from 10 to 50 Gbps, KV communication accounts for 16%–60% of job completion time (Fig. 1), with the share rising monotonically as bandwidth tightens — bandwidth itself is the first-order variable. And in an end-to-end decomposition on long-context workloads (Qwen2.5-32B, 2WikiMQA/HotpotQA), the Default configuration spends 82%–90% of total JCT on communication, dropping to 6–9% after compression (Fig. 15) — long context pushes the share to another order of magnitude. Two dimensions, one conclusion: the tighter the bandwidth and the longer the context, the more KV dominates.

The migration of the problem domain says something in itself: the network bottleneck of LLM inference has moved from connecting GPUs to moving state. (The fifth paper in Session 1, TurboBus, pools PCIe bandwidth — a scale-up interconnect problem we return to in Piece 4.)

2. What the Papers Actually Do

First, the problem definitions side by side. All six share the same backdrop (after PD disaggregation, KV must flow across the network), but they cut in at entirely different layers:

(Video-codec-KV and KVCodec are the same work's conference naming; this piece uses KVCodec throughout.)

2.1 KVServe: compression is not an algorithm choice, it is a state-dependent policy choice

After PD disaggregation, KV cache must move across the network, and compression is the first reflex for saving bandwidth. But how do production systems pick a compression method? Today's answer: pick one before launch and never touch it again. KVServe (ICT CAS, first paper of Session 1) opens by dismantling this default with three measurements. First, there is no cross-workload winner: KIVI (2-bit quantization) is most accurate on Qasper and near the bottom on GSM8K/HumanEval; DuoAttention (pruning) is exactly the reverse; CacheGen's compression ratio runs 6.20x on Multi-News and only 3.98x on HumanEval. Second, the optimal policy drifts with bandwidth: CacheGen (high ratio) wins at low bandwidth, MixHQ takes over mid-range, KIVI (low (de)compression overhead) reclaims the lead at high bandwidth; past their respective thresholds (50/55/110 Gbps), all three static methods are slower than no compression — (de)compression cost eats the communication savings. Third, negative optimization is real: on short-context tasks (GSM8K, HumanEval), static compression baselines post worse JCT than the uncompressed Default. Together, the problem is redefined: the compression config is not an offline constant; it is an online decision drifting with load and bandwidth.

KVServe's answer reframes "choosing a compression method" as constrained online selection: compression policy = workload type x effective bandwidth x SLO/quality budget. Three engineering moves. Decompose CacheGen/KIVI/KVQuant/MixHQ into pluggable components so the policy space is composable — this is the system's foundation. Offline, Bayesian profiling compresses the policy-space search from 1,000 hours to the 20-hour scale, producing a latency x accuracy x ratio Pareto frontier. Online, an analytical latency model issues baseline decisions (<1ms each), with a lightweight bandit correcting offline-online drift.

Validated across three workload families (QA/math/code) x three hardware tiers (10/50/100 Gbps) x four static baselines: state-aware policy selection never loses to the best static baseline at any test point, and beats every fixed choice when load or bandwidth drifts. Static optimum is only optimal in the state window it was chosen for.

Effectiveness holds, with clear edges: 20-hour-scale offline profiling is a real cost for services with frequent model updates, and the bandit depends on online signal quality. One finding deserves its own record: CacheGen ported to Qwen2.5 collapses in accuracy (HumanEval down to 57.32%), because Qwen2.5's K/V projections carry bias terms — non-zero-centered, asymmetric distributions that uniform quantization mismatches. Portability of compression methods is constrained by model-architecture details; this class of "architecture coupling" will recur across more models. The principle: compression policy must become runtime-native, the same evolution congestion control took from static configuration to BBR.

2.2 KVCodec: the idle video-codec ASIC

The other end of compression: where decompression runs. Both existing routes have hard flaws. CacheGen accelerates decompression with CUDA cores, but the SMs compete with inference — measured prefill time +50%, decode +20%, with peak decompression memory at 2.7x raw KV. ShadowServe moves decompression to a SmartNIC, but dedicated hardware blocks adoption. The production motivation is concrete: Kimi's production workload has 50% reusable KV (Mooncake paper data), so fetch-path decompression overhead accrues every day.

KVCodec's (Nanjing + Tsinghua AIR) insight is simple and effective: the NVENC/NVDEC video-codec ASICs on modern GPUs sit completely idle through the entire LLM inference process. Recruit them for KV coding: reuse H.265's lossless stages (intra/inter redundancy elimination), skip the lossy DCT+quantization, add token-sliced continuous-frame layouts — roughly 10x lossless compression; decompression runs on the NVDEC pool, physically isolated from CUDA cores, near-zero interference.

The prototype builds on LMCache v0.3.7. Results come in two layers: fetch-request TTFT (time to first token) drops 13.63x versus full prefill, 3.51x versus raw KV transfer, 1.52x versus CacheGen. More notable is the side effect: non-reuse requests see TTFT drop 77% too. Once decompression stops competing for CUDA, even requests that never touch KV reuse get faster.

Effective, with mid-term deployability: it needs driver-level NVENC/NVDEC integration, not a config change. The "idle dedicated hardware is a first-class resource" idea generalizes — AMD's AMF likewise; whether other inference-idle ASICs (JPEG decoders, display engines) can be used is the paper's open question.

2.3 DualPath: agentic workloads overturn the prefill-centric assumption

PKU + Tsinghua + DeepSeek-AI. DeepSeek's name appears on a SIGCOMM author list for the first time, in exactly this direction — the storage-I/O side of agentic inference. That a company known for model capability brought a production-inference networking problem to a networking venue is itself a signal worth recording.

It attacks a waste hidden by the default architecture. Motivation data comes from production agentic RL training traces: average 157 interaction rounds, 32.7K-token contexts, only 429 new tokens appended per round — 98.7% KV hit rate. Under such loads, inference barely does prefill computation; it mostly loads KV: DeepSeek-V3.2's cache-to-compute ratio reaches ~22 GB/PFLOP. The bottleneck moved decisively from compute to storage I/O. Yet the standard architecture forces all storage I/O through the prefill side: hit KV loads only via the prefill engine's storage NIC, saturating PE while DE idles. The bandwidth asymmetry is being wasted, not scheduled.

DualPath's answer is dual-path loading: alongside "storage -> prefill", add "storage -> decode -> RDMA across the compute network into prefill", turning idle decode-side bandwidth into a loading lane. Two companions: CNIC-centered traffic management with IB virtual-lane arbitration at 99:1 (model communication high priority, KV transfer low-but-starvation-free), and two-level scheduling that pools every node's storage bandwidth into one schedulable resource. The theory side derives a bottleneck-free P/D safety band: 17 <= P/D <= 72 (analytical, g=8, s=1), covering most production configs.

Experiments on DS 660B: offline JCT up to 1.87x, online throughput averaging 1.96x. Read the numbers knowing the baseline: the comparison is against DeepSeek's internal framework; the paper itself flags comparisons with public systems (SGLang+Mooncake) as not directly fair due to implementation differences. One discussion matters for the domestic stack: the design ports to RoCE via TC+DSCP, and the paper names UnifiedBus (Lingqu) and Ultra Ethernet QoS mechanisms as directly supportive of DualPath's needs. Mechanistically, this is the most portable of the six for domestic supernode stacks.

2.4 Connex: endpoint mobility as a first-class primitive

The missing communication layer of elastic inference. On worker join/leave/migrate, NCCL's communicator needs a global rebuild — communicator teardown, rank reshuffling, barrier synchronization — multi-second stop-the-world. The vLLM community already lists absent handover as a fundamental blocker of elastic inference. Elastic scaling, failure recovery, multi-tenant scheduling in production are all stuck on this layer.

Instead of patching each churn type, Connex (SIAT CAS + UPenn + Macau) defines a mobility contract: the semantics the communication layer must guarantee when join/leave/migrate fires. Three mechanisms each cover one leg: epoch routing turns reconfiguration into local route updates, no global rebuild; explicit handover uses sequence numbers + idempotent delivery + selective retransmission to keep migration local to adjacent stages; credit backpressure plus traffic classification keeps churn interference off the latency-sensitive paths.

On a 5-node A40 testbed under churn: P99 spike -85%, sub-second switches, steady-state overhead <5% (TTFT +10ms, TPOT (time per output token) +2ms). The mechanism holds; the limitation is explicit — 5 nodes is the main shortcoming, unverified at scale.

What is worth recording is the construction: all three components are classic distributed-systems techniques (epochs, idempotent delivery, backpressure), recomposed into a contract for GPU tensor streams with token-iterative semantics. Not a new wheel — known-correct mechanisms moved under new semantics. Deployment tracks the maturation of UCCL/NCCL replacements; mid-term.

2.5 Turbo: the aggregation operator moves into the switch

The bottom layer of the KV stack: not just moving KV, but moving the attention aggregation itself into the switch. The backdrop is sequence-parallelism's Pass-Q pattern: the master broadcasts a query, workers compute local attention, the master concentrates the aggregation. As sequences lengthen, the master becomes both a compute and a bandwidth bottleneck, plus incast. "Computation follows data", the old problem, in its attention-era version.

The Southeast University + PKU + Tsinghua + NUS team offloads both broadcast and aggregation to a Tofino2 programmable switch. The highest-value piece of the three is online lookup-table aggregation: softmax's exp-sum-division has no floating-point support on a switch, so Turbo decomposes the global reduction into pairwise operations, approximating nonlinear functions with an 11-bit-indexed, 2KB lookup table at negligible precision loss. This pairs with a rolling-forward scheme resolving RMT's (reconfigurable match-action, the architecture of Tofino-class chips) one-way no-writeback pipeline, and a load-aware aggregation tree (integer-linear-program ILP reduced to max-flow).

2,352 lines of P4 on Tofino2; 4-node physical testbed plus NS-3 large-scale simulation. TPOT up to -37% (8-GPU span contexts, vs Ring-Attention); -99% latency in simulation. The economics are clean too: the programmable-versus-commercial switch price gap is about $4,000 — a tenth of an H100 — and the whole switch draws 268.8W against a single H100's 700W TDP.

Effective, with two edges flagged in the evidence table: -99% is an NS-3 simulation figure (the physical testbed stops at 4 nodes; measured -37%); and on hardware, Tofino2's 20 physical stages are fully packed — 2 for gamma logic, 64-element attention segments fill the rest at ceil(64/4)+2 = 18 stages (16 tiling + 2 rolling window) with no headroom. Post-Tofino, the route's next bearers are commercial programmable silicon (the paper names Trident as portable; Centec sits in the EPIC working group).

2.6 Artic: the other end of AI traffic

The first five papers are about AI eating bandwidth; Artic is about the reverse: AI does not need that much bandwidth. AI Video Assistants (Grok, Gemini Live, ChatGPT, Copilot, Doubao) have an upstream-heavy pattern — high-bitrate video up, MLLM understanding in the cloud, audio back. Upstream-dominant, the directional mirror of KV's downstream dominance. But the entire RTC stack was designed for "humans watching video", its congestion control aimed at filling the pipe; when the consumer changes from the human eye to an MLLM, the machinery mismatches at the root. PKU's paper sits in session 18, but on traffic direction it is the dual of the KV line.

Two findings carry a new mechanism. First, the MLLM bitrate-accuracy curve saturates (around 968 Kbps; beyond that, accuracy flatlines). For an AI consumer, bandwidth headroom is worth more than bitrate — filling the pipe actively hurts. Second, the MLLM can feed back in real time "which regions the answer needs": the client allocates QP by that feedback, importance recognition on the server side, zero client overhead.

The accompanying benchmark DeViBench is auto-constructed (1,968 samples, $48.06 total), with experiments replaying 5G uplink traces. At equal accuracy, standard encoding needs 3,171 Kbps; context-aware encoding needs 908 Kbps. Overall accuracy +15.12%, latency -135.31ms.

The direction matters more than the numbers: AI-consumer traffic needs its own transport semantics — saturation-aware rate control, server-driven region priority — and this will rewrite the RTC layer's protocol design from the other side.

3. Route Analysis: A Map of the Technical Stack

Placed side by side, the six papers reveal a cleanly layered stack. The solution space of KV transfer decomposes along "what gets changed" into five layers:

Three verbs redistribute across the five layers:

• Save (compression): policy + encoding. KVServe proves static compression has negative-optimization regions; KVCodec proves codec compute can be zeroed out on idle ASICs. The ceiling of this route is set by attention's sensitivity to quantization — both works sidestep the "higher ratio" arms race and solve "when to compress, and where."
• Schedule: the architecture layer. DualPath treats bandwidth asymmetry as a schedulable resource; Connex treats membership change as a communication primitive. Both assume topology knowledge — scheduling quality is bounded by knowledge of network state.
• Move (transfer and compute): the in-network layer. Turbo's lookup tables push the boundary of "what a switch can compute" once more. The constraints are switch resources (Tofino2's 20 physical stages fully packed) and silicon generations (post-Tofino portability).

The five layers are not mutually exclusive — they are highly composable. KVServe's policy layer could in principle drive KVCodec's encoder; DualPath's path scheduling and Turbo's in-network aggregation act on different planes. The adjacency of Sessions 1 and 5 on the program itself hints at this stack.

4. Evidence Check

All numbers in this piece, anchored:

Two caveats stated as in the papers: Turbo's −99% is simulation, not testbed (4 nodes only); DualPath's baseline is DeepSeek's internal framework — the comparison to public systems (SGLang + Mooncake) is flagged by the authors themselves as not apples-to-apples. Neither is extrapolated here.

A third note: Turbo names Broadcom Trident (Trident 5) as a migration target, but it is not the only commercial RMT bearer — EPIC's author list includes Centec. "The only commercial option" would be false; this piece does not claim it.

5. Technical Assessment and Predictions

1. KV transfer will spawn its own protocol family rather than being absorbed by existing ones. None of the six papers modifies TCP/QUIC/RDMA itself — all work at KV's semantic layer (request-level profiles, layer-block formats, epoch routing, ATTN headers). Consistent with our earlier judgment: state transport needs semantics-aware transfer that generic protocols cannot provide. Prediction: within two years, a KV-first transfer protocol proposal will surface (likely under a UEC/Ultra Ethernet extension group); DualPath already names UE QoS support.

2. The compression battleground shifts from compressing smaller to selecting faster. KVServe's 50× search speedup plus <1ms online decisions establish the direction: lossless-ratio gains are diminishing (~10× is near the practical ceiling of H.265 lossless mode), while the payoff of correct selection has just opened. Runtime compression policy will become standard in inference serving — what adaptive bitrate became for video.

3. In-network aggregation and KV transfer will converge on the decode side. Turbo's in-switch attention aggregation and DualPath's KV-load scheduling act on the same stage (decode) in different planes (compute vs I/O). When a switch does both, bitmap/table resources and the aggregation tree's port binding become new coupling points. EPIC and HyNA sit directly on the in-network aggregation line; PReCCL and UBEP orbit it — Piece 2 expands.

4. Agentic load is the prime mover of KV's networkization, and it is broadly underestimated. DualPath's trace shows a 98.7% hit rate — agentic multi-turn KV reuse is categorically different from chatbot traffic. As agents go from tens to hundreds of turns and contexts toward 1M tokens, KV transfer volume scales another order of magnitude. DeepSeek bringing this production observation to SIGCOMM means it is not an academic hypothesis.

5. Switch-side attention aggregation is bounded by silicon cycles. Turbo packs Tofino2's 20 physical stages — 2 for gamma logic, 64-element segments at ⌈64/4⌉+2=18 stages (16 chunking + 2 rolling window), no headroom left — trading pipeline resources to the physical limit for a 37% TPOT cut. With Tofino discontinued, Broadcom Trident and Centec-class programmable silicon remain candidate bearers; the route's future depends on the next generation's SRAM/ALU budgets. On a five-year horizon, wafer-scale networks and DPUs are the likelier substrate.

6. Deployability in three tiers. Immediately copyable: KVServe's policy layer (vLLM integration open-sourced) and DualPath's 99:1 VL arbitration (a configuration change on any IB cluster). Mid-term: KVCodec (driver-level NVENC/NVDEC integration; prototype on LMCache v0.3.7) and Connex's mobility contract (maturing alongside UCCL-class NCCL replacements). Long-term research: Turbo's fleet-wide in-network aggregation (bounded by programmable-switch installed base). For Chinese supernode stacks (UnifiedBus ecosystem), DualPath is the most directly portable design — the paper provides the QoS mapping.

---

Disclosure: This piece draws on the full SIGCOMM 2026 versions of six papers (ACM DL; DOI prefix 10.1145/3789240): KVServe, DualPath, KVCodec, Connex, Turbo, Artic. All multiples and percentages follow the papers' own wording, with per-claim experimental conditions in the evidence table; Turbo's -99% is an NS-3 simulation figure (-37% measured on the 4-node testbed), and DualPath's baseline is DeepSeek's internal framework (not directly comparable to public systems, per the authors). Data as of August 22, 2026.

Piece 1 of the SIGCOMM 2026 close-reading series. The overview is Piece 0. Next: "Collectives Become a Runtime" — how fifteen papers rewrite the communication stack for the MoE era.