KV Cache as Infrastructure: When the Cache Layer Decouples from the Inference Engine

Between June and July 2026, KV Cache is transforming from an in-engine memory management optimization into a new infrastructure layer that GPU vendors, storage vendors, and the open-source community are simultaneously betting on. Storage primitives, transport protocols, scheduling and routing, commercial products, and hardware offload are five tracks landing in rapid succession within six weeks. Who will define this layer, and how deep into the physical media will it sink?

1. What Happened in Six Weeks

Before May 2026, the KV Cache story lived entirely inside inference engines. vLLM's PagedAttention managed KV block lifecycles within GPU HBM; SGLang's RadixAttention performed prefix sharing within a single worker. These optimizations did not cross single-machine boundaries.

On May 7, vLLM's official blog published a deep-dive on Mooncake Store integration—on 60 GB200 GPUs, Mooncake's distributed KV pool achieved 3.8× throughput improvement, 46× TTFT improvement, and 8.6× end-to-end latency improvement. This was the first time KV Cache appeared as "independent infrastructure" in a production-grade benchmark.

Over the following six weeks, changes landed densely across five layers:

Storage layer: from in-engine to independent storage backends. vLLM's native OffloadingConnector evolved to support a three-tier architecture (GPU HBM → CPU DRAM → filesystem/object storage). LMCache integrated Ceph, NetApp ONTAP S3, and Redis as remote storage backends. llm-d (jointly developed by IBM/Red Hat/Google/Alibaba, entered CNCF Sandbox in March 2026) introduced FS Backend, leveraging POSIX filesystem shared semantics to natively support cross-node KV reuse. AWS SageMaker began offering one-click KV offloading deployment.

Transport layer: two routes to production simultaneously. Mooncake Transfer Engine took the RDMA GPUDirect zero-copy path, already integrated into vLLM, SGLang, and TensorRT-LLM. LMCache v0.5.0 took the CPU DRAM P2P sharing path, introducing HiddenStateStore. NVIDIA's NIXL (NVIDIA Inference Transfer Library) emerged as a third force—announced at GTC 2025, version 1.3 in July shipped with a built-in DDN Infinia storage plugin, distributed with the Dynamo container.

Scheduling layer: from hash routing to ML-predicted routing. Ray 2.56.0 (June 30) promoted PrefixCacheAffinityRouter to a formal release. llm-d's cache-aware scheduler integrated deeply with Kubernetes Gateway API Inference Extension. GKE Inference Gateway went further—introducing an XGBoost model to predict TTFT and TPOT per request, using ML for routing decisions.

Commercial layer: storage vendors collectively entering. DDN Infinia became the first storage product natively integrated into NVIDIA NIXL (shipping with Dynamo containers starting July). VAST Data partnered with CoreWeave to benchmark Dynamo KV offload: GPU efficiency improved 90%, TTFT improved 20×. NetApp ONTAP S3 integrated with LMCache. Xinnor used Lustre RO PCC + xiRAID to bring shared-storage KV Cache latency close to local NVMe.

Hardware layer: AFD lands, LPU joins the ranks. NVIDIA's Vera Rubin platform introduced AFD (Attention-FFN Disaggregation)—GPU handles attention, LPU handles FFN, the decode computation pipeline is physically split across two chip types. The LPX rack (256 Groq 3 LPUs) provides 40 PB/s of aggregate SRAM bandwidth for FFN computation, far exceeding HBM bandwidth.

Six weeks ago, these components were scattered across different project roadmaps. Six weeks later, they form a complete technology stack. But understanding this stack requires starting from a more fundamental question: what is KV Cache, really—its physical characteristics dictate all the architectural choices above.

2. Physical Characteristics of KV Cache

KV Cache is not generic data. It has a distinct set of access patterns that dictate what the software architecture and hardware design should look like. Without understanding these physical characteristics first, every subsequent analysis is built on sand.

Block Size

The basic unit of KV Cache is the KV block—a group of tokens' Key and Value matrices. Take a typical trillion-parameter MoE model (80 layers, 8 KV heads (GQA), 128 head dim, FP8 precision):

• Per-token per-layer KV: 8 heads × 128 dim × 2 (K+V) × 1 byte (FP8) = 2,048 bytes
• Per-token across all 80 layers: 2,048 × 80 = 163,840 bytes ≈ 160 KB
• A 16-token block: ~2.5 MB
• A 256-token block: ~40 MB

KV block access granularity is at the MB scale—not CPU cache line (64B), not file (GB scale). This dictates what granularity the addressing mechanism should operate at.

Write Pattern: write-once, append-only

The prefill phase writes all blocks at once; after that, no modifications. The decode phase appends one new block per token. The overwhelming majority of KV blocks are written exactly once in their lifetime.

This is fundamentally different from CPU caches (which may be repeatedly modified) and databases (random updates). write-once means the coherence mechanism doesn't need to track "who modified this data"—once written, it's immutable.

Read Pattern: full scan + cross-session random

During decode, generating each token requires reading the entire history of KV blocks for the current session. The read is a full scan—not grabbing a few random blocks, but traversing the session's entire KV.

But from a cross-session perspective, the scheduler decides which session's KV to read—so from the global view, reads are random.

Lifecycle: seconds to minutes

Created at session start, deleted at session end. The vast majority of KV Cache lives for seconds to minutes. No persistence—if the session drops, the KV is useless.

This is completely different from what storage products assume. Storage systems are designed for "data must not be lost" (persistence, replication, EC). KV Cache can be recomputed if lost—the cost is tens of milliseconds to seconds of prefill time.

Sharing Pattern: multi-reader, single-writer

Common prefixes (system prompts, document context) are shared across sessions. One producer writes, N consumers read. Cross-session sharing of prefix KV blocks is the core value driver of KV Cache infrastructure—the 94.2% cache hit rate measured in agentic scenarios demonstrates that this sharing pattern is not an edge case, but the mainstream workload.

Coherence Requirement: session-scoped

Strong consistency within a session: the KV read during decode must include all blocks appended by previous decode steps. Reading incomplete KV will directly cause the model to generate incorrect tokens.

No cross-session consistency needed: different sessions' KV is independent, and parallel reads/writes don't interfere.

Bandwidth Requirement Estimation

A inference cluster with 128K context and 1,000 concurrent sessions:

• Total KV Cache: 1,000 × 128K × 160 KB/token ≈ 20 TB
• Per-token KV read bandwidth per session during decode: 128K × 160 KB / decode_step_latency. If decode step = 50ms, per-session read bandwidth ≈ 400 GB/s
• 1,000 concurrent sessions decoding simultaneously: aggregate read bandwidth demand ≈ 400 TB/s

For comparison: an 8×Rubin GPU node has approximately 176 TB/s of aggregate HBM bandwidth. A single node's HBM bandwidth is insufficient—the KV Cache read bandwidth demand exceeds single-machine capacity by approximately 2.3×. Adding GPUs increases bandwidth, but each GPU node also adds a full suite of compute capacity and cost, while KV Cache only needs bandwidth, not proportionally more compute.

This number explains why KV Cache must be shared across nodes: not because a single GPU cannot hold all the KV (though it can't), but because a single GPU's HBM bandwidth cannot feed decode's read demands.

Fundamental Difference from Training Storage

Training is write-intensive long sequential IO—gradient updates and checkpoint saves are large-block sequential writes. Inference is massive quantities of small-granularity random read/write—KV block-level load/save, frequent but small. The market logic for training storage (high capacity, high bandwidth, sequential optimization) does not fully apply to inference storage. What inference needs is: low-latency random access, high-concurrency small IO, and block-level lifecycle management.

3. The Four-Layer Software Landscape

The software stack for KV Cache infrastructure formed a four-layer structure within six weeks. Each layer has 2–3 production-grade implementations with different design philosophies.

Storage Primitive Layer (where KV lives):

• vLLM native multi-tier offloading (RFC #38260): engine-embedded. TieringManager with three tiers (HBM → DRAM → FS/PD peer), KV blocks cross-TP normalized to TP=1 canonical form.
• LMCache: independent cache middleware. Three-tier storage (HBM → DRAM → NVMe/Redis/S3), v0.5.0+ MP mode runs as an independent daemon. The LMCache team participates in maintaining the vLLM Production Stack. The academic paper (arXiv 2510.09665v2) is the first to systematically define the boundary between offloading and PD disaggregation.
• Mooncake Store: distributed KV pool. RDMA GPUDirect path, KV blocks move directly from GPU HBM to remote DRAM/SSD. FAST 2025 Best Paper disclosed: Kimi's production environment with thousands of nodes processing 100 billion tokens daily, A800 nodes handling 115% more requests, H800 nodes handling 107% more.
• llm-d FS Backend: zero-dependency path. KV blocks written to any POSIX filesystem, shared semantics natively support cross-node reuse. Entered CNCF Sandbox in March 2026. v0.4.0 (December 2025) achieved 40% latency reduction per output token on DeepSeek V3.1 + H200.
• NVIDIA Dynamo KVBM: GPU vendor path. Built-in KV Block Manager, three-layer architecture (LLM runtime → logical block management → NIXL transport), native KV-aware routing integration.

Transport Protocol Layer (how KV moves):

• RDMA school (Mooncake Transfer Engine / NIXL): GPUDirect zero-copy, Multi-NIC pooling, topology-aware. Integrated into vLLM, SGLang, TensorRT-LLM.
• P2P CPU Memory school (LMCache): direct CPU DRAM sharing. Measured on 32 × 10K-token long-context scenarios: TTFT improved 4×, round time improved 5× (LMCache blog, 2026-06).
• Filesystem path (llm-d): POSIX semantics guarantee consistency, latency is an order of magnitude higher than RDMA, but cross-domain sharing is natively supported. Xinnor demonstrated that with Lustre RO PCC + xiRAID, shared storage can approach local NVMe latency.

Scheduling & Routing Layer (which node gets the request):

• Ray 2.56.0 PrefixCacheAffinityRouter: two-tier strategy (prefix tree routing under load balance, falls back to Power of Two Choices when imbalanced). Limitation: only routes within ReplicaGroup.
• Mooncake Conductor (RFC stage): "sandwich architecture," Conductor scheduling layer sandwiched between inference engine and KV Store.
• llm-d cache-aware scheduler + GKE Inference Gateway: K8s Gateway API integration. GKE introduced XGBoost ML-predicted routing.
• NVIDIA Dynamo KV-aware routing: paired with KVBM, supports both aggregated and disaggregated modes.

Operational Governance Layer (who manages multi-tenancy and SLAs):

Mooncake Store's three-piece suite (tiered scheduling, tenant quotas, high availability) was the earliest signal. vLLM's Hybrid KV Cache Manager manages HBM internal resource allocation. Dynamo's governance is tied to the NIXL plugin ecosystem.

4. Interface Contract: Where the Engine Ends and the Cache Layer Begins

The four-layer landscape answers "who is doing what." But a more fundamental question remains untouched: where exactly is the interface boundary between the KV Cache infrastructure layer and the inference engine? Which responsibilities belong to the engine, and which can be carved out?

The Current de facto Standard: vLLM KVConnector Interface

vLLM's KVConnectorBase_V1 is the only widely implemented interface contract. It splits KV Cache management into two sides:

Scheduler-side—the engine asks the cache layer: "How much of this request's prefix do you already have?"

• get_num_new_matched_tokens() returns the number of tokens matched in the external cache
• The engine uses this to skip prefill computation for the already-cached prefix
• Match logic is defined by the cache layer (exact prefix hash, semantic matching, etc.)

Worker-side—the cache layer moves KV data into the engine's HBM:

• start_load_kv() asynchronously preloads before the request arrives
• wait_for_save() asynchronously writes back after the request completes
• KV data ultimately lands in vLLM's PagedAttention-managed HBM pages

The clarity of this interface: the engine owns the KV data format and HBM-internal management; the cache layer owns cross-instance storage, transport, and matching. The contract between them is defined by two atomic operations: one read (load), one write (save). Everything in between—the storage medium, transport protocol, eviction policy—is the cache layer's black box.

Three Tensions at the Interface Boundary

Tension 1: TP normalization. When KV blocks are offloaded from GPU HBM to secondary storage, the KV shard format differs across tensor parallelism ranks. The cache layer must normalize to TP=1 canonical form in secondary storage—meaning the cache layer needs to understand the engine's tensor parallelism, even though it doesn't do computation. The cache layer is not a pure "store and retrieve" role; it needs to know the model's geometric shape across the GPU cluster.

Tension 2: Semantic reuse. vLLM RFC #44223 (filed in 2026) proposed "semantic KV Cache reuse"—not requiring exact prefix matching, allowing approximate or cross-request KV reuse. When the same document is referenced by different prompts, existing KV cache can be partially reused. But this requires the cache layer to make matching decisions, and matching correctness directly affects inference quality. If semantic matching goes wrong, the model generates tokens on incorrect attention states, with unpredictable results. This RFC is currently conservatively scoped to "prefix-anchored continuous reuse," but it opens a direction: the cache layer may evolve from a "storage service" to an "inference-quality-related service."

Tension 3: Mixed attention models. When a model uses both full attention and sliding-window attention simultaneously (e.g., Mistral, Gemini), different layers' KV cache have different lifecycles. Sliding-window layers' KV can be discarded beyond the window, while full attention layers' KV must be fully retained. The cache layer must understand the model's attention architecture to implement correct eviction policies—this is not something simple LRU can cover.

Functional Requirements for an Independent KV Cache Product

From the tensions above, we can derive the complete capability checklist for an independent KV Cache product:

The last two rows are the biggest gaps. All current solutions assume the same engine (vLLM → vLLM, SGLang → SGLang). If a Mooncake Store instance simultaneously serves vLLM and SGLang workers, KV format compatibility is unverified. This is not a technical detail—it is the fundamental question of whether "KV Cache infrastructure" can truly be independent of the inference engine.

5. Six Challenges of Multi-Node Cluster Deployment

All analysis up to this point has stayed at the single-instance or small-cluster perspective. But production inference clusters are multi-node, multi-replica, and span multiple switch layers. When KV Cache infrastructure is deployed into real multi-node clusters, six hard problems erupt simultaneously.

Challenge 1: Topology Awareness—Which Node Holds the KV Cache

A typical PD-disaggregated inference cluster: P cluster (Prefill) with 8–32 GPU servers, D cluster (Decode) with 16–64. Network is Spine-Leaf or Fat-Tree.

KV Cache is produced at P nodes and needs to be consumed by D nodes. P has 32 nodes, D has 64—which P's KV goes to which D?

Quantifying topology cost: Assuming P nodes and KV pool are under the same Leaf, RTT ~1μs, bandwidth 400 Gbps (50 GB/s). Cross-Spine: RTT ~3–5μs, bandwidth limited by spine convergence ratio (possibly 100–200 Gbps). A 128K-context KV Cache of ~10–25 GB takes 0.2–0.5 seconds to transfer within the same Leaf, 0.5–2.5 seconds cross-Spine.

Physical conditions for cache value: Transfer latency < prefill recomputation time. For 128K context, prefill recomputation takes ~2–5 seconds (depending on GPU model and model size). Cross-Spine KV fetch taking 1 second → worthwhile (faster than recomputing). For 4K short context, prefill recomputation takes ~30–50ms → cross-Leaf KV fetch taking 100ms is worse than recomputing.

Core problem: A topology-unaware KV Cache pool scatters one session's KV across the entire cluster. Mooncake Conductor attempts to solve this in its paper but is still at RFC stage. LMCache MP currently does not do topology awareness.

Challenge 2: Consistency—Session Migration Race Conditions in Multi-Round Agentic

Multi-round agentic scenario: Round 1 decodes on D#7, Round 2 the scheduler migrates the request to D#12 (because D#7's load increased). D#7's write-back of Round 2 KV may not have finished flushing before D#12 starts reading.

Consequence of reading incomplete KV: the model generates tokens on incomplete attention states, with unpredictable output quality.

Current approaches: LMCache uses LRU + async writes, with weak consistency guarantees—on cache miss, it falls back to recomputing prefill; correctness is backstopped by "recomputation." Mooncake Store uses a master for version management, but the consistency model is not detailed in the paper.

What's needed is session-scoped read-after-write consistency—guaranteeing the latest KV is read within the same session, with relaxed requirements across sessions. This is not global strong consistency (too heavy), nor eventual consistency (too weak).

Challenge 3: Failure Domain—What Happens When the KV Pool Goes Down

Once an independent KV Cache pool becomes a critical dependency in the inference chain, its availability directly affects inference service.

Failure consequence estimation: Assume the cluster normally relies on a 90% cache hit rate (typical agentic scenario). KV pool failure drops hit rate to 0%. The prefill cluster's load = the original 10% prefill work + absorbing the other 90% of prefill → load spikes 10×. If the hit rate is 94.2% (Codex measured), load spikes 17×. The P cluster doesn't have 10–17× surplus compute; it immediately queues or rejects service.

Redundancy cost: The KV Cache pool needs replicas. 1,000 concurrent sessions × 50GB KV/session = 50TB pool total, three replicas = 150TB memory/SSD.

Degrade, don't recover: The cost of KV Cache loss is recomputing prefill (seconds), not data loss. So the failure recovery strategy can be "degrade"—pool goes down, fall back to no-cache mode. But this requires the P cluster to have surplus compute to absorb the burst of prefill. Without surplus, failure cascades.

Challenge 4: Network Bandwidth Economics—The Cost Crossover Point of Transfer vs. Recomputation

Bandwidth requirement worked example: A cluster with 128K context × 1,000 QPS. Each prefill produces ~25GB of KV Cache (FP8). On cache miss, 1,000 prefills/second generate 25 TB/s of KV transfer demand. On 400 Gbps (50 GB/s) InfiniBand NDR, this would require 500 links to absorb—physically impossible.

Cost crossover analysis:

The break-even point for cache hit rate depends on three variables:

1. Recomputation cost = prefill GPU compute time × GPU hourly price
2. Transfer cost = KV block size × network transfer time (including queuing)
3. Storage cost = KV block storage footprint × storage unit price × lifetime

Using 128K context, H100 GPU ($2.5/hour), 400Gbps network:

• Single prefill recomputation: ~3 seconds × $2.5/3600 ≈ $0.002/request
• Single KV transfer (25GB / 50GB/s): ~0.5 seconds, network allocated cost ≈ $0.0003/request (based on network equipment cost amortization estimates)
• KV storage (25GB × DRAM $3/GB/month × 60 seconds lifetime): 25 × ($3 ÷ (30×24×3600)) × 60 ≈ $0.0017/request

Transfer + storage cost ($0.002) ≈ recomputation cost ($0.002)—costs are converging. This means the economic value of caching is not in the per-request cost delta, but in the hit rate. When hit rates are high, storage cost is amortized across many hit requests (each session stores one KV copy, reused by multiple requests), so the per-hit storage cost is far below $0.0017. When hit rates are low, storage cost cannot be effectively amortized.

The break-even point is approximately 50% hit rate. Above 50%, cache amortized cost is lower than recomputation—caching is valuable. Below 50%, recomputing is better. The 94.2% hit rate measured in agentic scenarios far exceeds the break-even point—cache has enormous economic value. But short-context scenarios with hit rates potentially below 50% require case-by-case economic analysis.

This analysis directly informs architecture choices: long-context + high-concurrency + agentic multi-round scenarios have high hit rates, making cache highly valuable. Short-context, low-concurrency scenarios have questionable cache economics. The target market for KV Cache infrastructure is long-context, high-concurrency inference scenarios.

Challenge 5: Version Lifecycle—KV Goes Stale on Model Updates

Model updates (fine-tuning, weight updates, architecture changes) → KV Cache generated by previous versions is unusable. Different models may have different numbers of attention heads, layers, and hidden dimensions.

More subtly, different checkpoints of the same model (post-fine-tuning) are technically KV-reusable, but semantically potentially inconsistent—the fine-tuned model's attention patterns may have shifted, making the old KV Cache's attention states inaccurate. In long-context scenarios, this can cause quality degradation.

KV Cache is version-bound temporary data, not a persistent asset. Every model update invalidates the entire KV Cache pool. This conflicts with storage products' "persistence" value proposition. The correct operational strategy is canary updates—new requests generate KV with the new model, old KV naturally expires through eviction, without bulk invalidation.

Challenge 6: Trust Boundary—Physical Isolation for Multi-Tenant Sharing

KV Cache is the complete context of a user's conversation with the model—containing prompts, conversation history, and tool call results. In a multi-tenant shared KV Cache pool scenario, "can tenant A see tenant B's KV blocks" is not just a quota issue; it's a data security issue.

Side-channel risk: In a shared memory pool, an attacker could theoretically infer other tenants' KV block access patterns by measuring cache access timing—cache timing side-channel. Mooncake's tenant quota mechanism limits storage space allocation but does not prevent physical-level timing observation.

TEE limitations: GPU-side TEE (e.g., NVIDIA Confidential Computing H100) protects computation inside the GPU, but once KV Cache is offloaded to CPU DRAM or external storage pools, it leaves TEE protection scope. Offloaded KV blocks need their own encryption/isolation mechanism.

Current gap: No KV Cache infrastructure product provides hardware-level cross-tenant isolation. Mooncake's tenant quotas are software-level logical isolation. If KV Cache is to truly become a cross-tenant shared infrastructure layer—trusted by different customers like cloud object storage—the trust boundary must evolve from software logical isolation to physical-level guarantees.

What the Six Challenges Point To

The common root of these six challenges points in one direction: the current compute-memory architecture assumption—GPU does all computation, PCIe/network handles all connectivity, software manages all tiers—has been punctured by KV Cache's demands. Pure software solutions can mitigate but cannot fundamentally solve the problem. Every KV block load/store traverses software-managed memory hierarchies, and every hop adds latency and uncertainty.

The fundamental solution requires rethinking KV Cache's position in the memory hierarchy at the hardware level.

6. Reasoning from Workload to Ideal Hardware

This is the most technically dense chapter. Methodology: starting from the physical characteristics in Chapter 2 and the six challenges in Chapter 5, derive what capabilities hardware should provide, eliminate unsuitable approaches, and reason toward the ideal form.

Methodology boundary statement: In this chapter, NVIDIA AFD, Groq 3 LPU specifications, and Mooncake benchmark data are publicly disclosed facts. KV Memory Node, GPU-side KV Cache Controller, and Memory Fabric are design-space reasoning based on physical constraints—not product predictions, but architectural direction analysis.

6.1 First, Eliminate the CXL/PCIe Path

CXL looks like a natural choice: open standard, cache-coherent, mature spec for memory disaggregation. Before analyzing whether it fits KV Cache, let's acknowledge its strengths:

CXL's theoretical advantages: Type-3 device (memory expander) cache coherence semantics naturally match KV Cache's write-once characteristic—KV blocks are rarely modified after one write, so Type-3 device coherence overhead (snoop, etc.) is nearly zero for this workload. CXL 4.0 provides 128 GT/s bandwidth and pooling capability. Samsung CMM-D measured performance approaches DRAM.

But CXL is built on the PCIe physical layer, and this DNA dictates three ceilings:

Ceiling 1: Latency. CXL 3.0 end-to-end latency ~200–500ns (including PCIe TLP encapsulation/decapsulation, CXL flit processing, switch forwarding). Compare: HBM ~10ns, NVLink ~50ns. On the decode hot path, each token requires KV reads across 80 attention layers (80-layer model). CXL latency accumulates within a single decode step: 80 × 300ns ≈ 24μs. For a decode step (20–50ms), this is 0.05–0.1%—acceptable per step, but noticeable under high concurrency.

Ceiling 2: Protocol semantics. CXL operates at cache-line granularity (64B) coherence. KV blocks are at MB scale (2.5–40MB). Using cache-line coherence to manage MB-scale data is like moving bricks with a microscope—the tool granularity doesn't match the task granularity. Massive coherence protocol overhead is spent where it's not needed.

Ceiling 3: Switch hierarchy. CXL goes through PCIe switch hierarchy. CXL 3.0's Fabric Manager allows port-based routing to partially bypass traditional switches, but per-hop flit processing latency (~50ns/hop) persists. Compared to NVLink bridge's zero-hop direct connection, multi-hop scenarios still accumulate latency. KV Cache's bandwidth demands require switchless direct interconnect—similar to NVLink bridge mode (GPU-to-GPU direct). PCIe's switched architecture is an IO device paradigm, not a memory paradigm.

Conclusion: CXL is not suitable as the hot-tier interconnect for KV Cache. It can have a place in the cold tier (cross-domain sharing, infrequently accessed KV blocks), but KV Cache on the decode hot path needs NVLink-level latency and bandwidth, not PCIe-level.

6.2 Deriving Hardware Requirements from Physical Characteristics

Chapter 2 defined six physical characteristics of KV Cache. Each characteristic derives a hardware requirement:

Physical characteristic → Hardware requirement 1: Block-level addressing (2.5–40MB granularity)

KV blocks are MB-scale; they don't need cache-line (64B) granularity coherence, nor filesystem inodes. What's needed is content addressing at block granularity—give the block an ID (token prefix hash), and hardware directly locates the physical position.

The cost of doing this in software: hash → metadata table lookup → physical address → DMA transfer, three levels of indirection. An index of 1 million KV blocks requires several GB of memory. If the memory controller hardware understands block semantics, a single access locates the data—similar to CAM but operating at MB granularity.

Physical characteristic → Hardware requirement 2: write-once coherence

99% of KV Cache blocks are write-once-read-many. Standard MESI/MOESI coherence protocols maintain state per cache line (Modified/Exclusive/Shared/Invalid) because cache lines may be arbitrarily modified in CPU multi-core scenarios. Maintaining MESI state for write-once data is pure waste.

What's needed is a partitioned coherence model: blocks being written obtain exclusive (preventing reads of incomplete data), drop to read-only shared after writing completes, and are no longer tracked afterward. Software uses hints (the done bit in the block header) to tell hardware when to switch states. This is an order of magnitude lighter than standard coherence.

Physical characteristic → Hardware requirement 3: Independent bandwidth scaling

The 400 TB/s KV Cache read bandwidth requirement cannot be met by adding GPUs—adding GPUs also adds compute (and cost). KV Cache doesn't need more compute; it only needs more bandwidth to read it.

What's needed is a memory-only physical node—no GPU, just large amounts of high-bandwidth memory + high-bandwidth interconnect ports. Pure bandwidth/capacity nodes, compute-free.

Physical characteristic → Hardware requirement 4: Session-scoped consistency domain

Standard distributed system global consistency is too heavy. Cross-session consistency isn't needed. What's needed is hardware-enforced session affinity—KV blocks of the same session are physically aggregated into one consistency domain, guaranteeing read-after-write within the domain, with zero coherence overhead across domains.

Software relies on the scheduler for best-effort session affinity, but sessions get migrated under load imbalance. Hardware partitioning can guarantee consistency through hardware mechanisms even during migration.

Physical characteristic → Hardware requirement 5: Embedded version control

Model update → KV Cache fully invalidated. If the memory controller embeds model_id in the block header, cache lookup automatically filters incompatible versions at the hardware level—no software overhead. Incompatible blocks are automatically reclaimed on the next write.

A very small hardware change (a few bits in the block header), but it eliminates an entire class of software-level race conditions.

6.3 The Hardware Form Derived from Reasoning

Five hardware requirements point to a complete hardware system:

Component A: KV Memory Node (a new data center node type)

Not a GPU node (no computation), not a CPU node (no scheduling), not a storage node (no persistence). A new category:

• Large-capacity high-bandwidth memory: TB-scale DRAM, bandwidth target 50–100 TB/s per node
• KV-aware memory controller: understands block semantics, content-addressed lookup, write-once coherence, model_id filtering
• Built-in compression engine: FP8/INT4 or custom compression, reducing bandwidth pressure
• High-bandwidth interconnect ports: ≥1 TB/s per link to GPU nodes, NVLink-level latency (~50–100ns)
• Independent failure domain: independent power, cooling, management plane
• Session partition: hardware support for dividing memory into multiple logical consistency domains

Physical form factor: 1U–2U rack-mount device, front panel packed with memory modules, no GPUs. Connected to GPU nodes in the same rack via high-bandwidth memory-semantic fabric.

Component B: GPU-Side KV Cache Controller (a module inside GPU silicon)

A new module sitting between the GPU memory controller and NVLink controller:

• Asynchronous prefetch engine: before decode needs a particular KV block, prefetch it from the KV Memory Node to HBM ahead of time
• Write-back buffer: after prefill completes, KV blocks first enter the buffer, then asynchronously flush to the KV Memory Node without blocking the inference pipeline
• Coherence partition manager: manages the coherence state of active sessions on this GPU, coordinating with the KV Memory Node

Component C: Memory Fabric (the interconnect layer)

The physical interconnect between KV Memory Nodes and GPU nodes. Not PCIe/CXL, but memory-semantic direct connect:

• Latency ≤ 100ns (approaching NVLink, far better than CXL/PCIe)
• Per-link bandwidth ≥ 1 TB/s
• Multi-port: one KV Memory Node simultaneously connects to 4–8 GPU nodes
• Partition capability: fabric switch supports grouping ports into multiple coherence partitions

Physical implementation candidates: NVLink extension (NVIDIA controls the standard, already has Grace CPU + GPU interconnect experience), or custom memory fabric. Not PCIe.

6.4 AFD and KV Cache Bandwidth Economics

NVIDIA's AFD (Attention-FFN Disaggregation) changes KV Cache's bandwidth economics—but it is not a direct KV Cache solution.

Decode bandwidth contention without AFD: during decode, for each token, GPU HBM bandwidth is split between KV Cache reads and FFN weight reads. Both compete for the same pipe—this is the "decode bandwidth wall."

Software mitigation approaches and their ceilings:

• Large batch: FFN weights amortized across tokens in the batch → but KV Cache capacity pressure explodes
• GQA: KV Cache compression → already standard, diminishing incremental gains
• FP8 KV Cache: bandwidth halved → usable but not a qualitative change
• Speculative decoding: 2–3× speedup → doesn't change the architectural bottleneck
• HBM bandwidth growth: H100→Rubin improves ~6.5× → consumed by context growth (4K→128K = 32×)

The software path can push back the bandwidth wall by 3–5×, but cannot dismantle it.

With AFD: the LPU's SRAM (~500MB, 150 TB/s) handles FFN computation. The GPU's HBM bandwidth is no longer consumed by FFN weights, leaving nearly all of it for KV Cache. KV Cache effective bandwidth improves 2–3× (depending on FFN's share of the original bandwidth budget).

But AFD amplifies KV Cache offload demand—faster decode → more concurrent sessions per GPU → larger KV Cache footprint → greater dependence on external storage pools. AFD optimizes compute-side efficiency; KV Cache infrastructure optimizes storage/transfer-side capacity. The two are complementary, not substitutable.

AFD only works for MoE models—it leverages Expert All-to-All communication channels to transfer activations. For dense models, AFD communication overhead is too high. This means AFD's value depends on model architecture trends.

6.5 Hardware Media Tiering

Media tiering after eliminating CXL:

Each tier's latency differs by 10–50×, dictated by physics. The 100× gap between HBM and RDMA is exactly what the NVLink Memory Node needs to fill. CXL originally aimed to fill this gap, but the PCIe physical layer's latency and switch hierarchy mean it can only land on the RDMA-leaning side between HBM and RDMA—not close enough to the GPU's hot path.

6.6 Conclusion: KV Cache Is a Memory Hierarchy Management Problem

KV Cache's characteristics (block-level granularity, write-once, session-scoped consistency, second-scale lifecycle, bandwidth-intensive) more closely resemble CPU's L1/L2/L3 cache → DRAM → swap hierarchy management than distributed storage.

Historically, memory hierarchy management has always been a matter of software-hardware co-design—CPU cache line size, TLB structure, and NUMA affinity are all cases where hardware defines the mechanism and software optimizes the policy. KV Cache is likely to follow the same path: today it's all software (LMCache/Mooncake); in the future, hardware will provide block-level addressing and session partition, while software retreats to the policy layer.

7. Competitive Landscape: A Three-Way Contest

Stepping back from hardware reasoning to reality—all current players are still doing software. But Chapter 6's analysis tells us this domain's endgame is software-hardware co-design. Which players hold tickets?

Route 1: Engine-embedded (llm-d / vLLM native). KV cache as the engine's OffloadingConnector module. llm-d entered CNCF Sandbox, K8s-native. Limitation: cross-instance caching is bounded by scheduling scope; storage and compute scaling are coupled.

Route 2: Independent cache systems (LMCache / Mooncake). LMCache independent daemon + Mooncake distributed KV pool. Strategic partnership announced in May 2025. Advantage: storage-compute decoupling. Limitation: extra serialization overhead; cross-project coordination.

Route 3: GPU vendor-led (NVIDIA Dynamo / KVBM / NIXL). Dynamo orchestration + KVBM built-in cache + NIXL transport + DDN/VAST storage plugins. NVIDIA controls the full stack. If the NVLink Memory Node reasoning from Chapter 6 is the right direction, only NVIDIA can build it (controlling the NVLink standard).

Who wins:

• Single cluster, NVIDIA hardware: Dynamo + KVBM. NVIDIA controls the full stack; third-party space gets compressed
• Mixed hardware, AMD/Intel: Mooncake + LMCache. Cross-hardware transport layer
• K8s-native: llm-d. CNCF endorsement, but lacks independent transport layer
• Cross-cluster/cross-cloud: filesystem path. RDMA is constrained by network topology; POSIX is natively cross-domain

The role of storage vendors: DDN went from selling an independent product to being the default NIXL plugin. VAST Data partnering with CoreWeave proved storage vendors can create quantifiable value at the KV Cache layer (GPU efficiency +90%, TTFT improved 20×). KV Cache storage won't become an independent product category, but it will change storage array selection criteria—ultra-low tail latency and KV-level small IO performance become hard requirements for inference scenarios.

8. Routing Layer Evolution

KV-aware routing completed three generational leaps in six weeks:

Generation 1: Hash affinity. The basic pattern in GKE/llm-d—hash the token prefix, look up which replica has cache. Simple and predictable, load-unaware.

Generation 2: Prefix tree + load awareness. Ray 2.56.0—prefix tree routing under load balance, falls back to Power of Two Choices when imbalanced.

Generation 3: ML-predicted routing. GKE predicted-latency-based routing—XGBoost predicts TTFT/TPOT in real time, combined with prefix-cache-affinity filter. The significance isn't "using ML," but that the objective function shifted from "cache hit rate" to "predicted latency." The two align most of the time, but diverge under load imbalance or hardware heterogeneity. ML routing handles the divergence; rule-based routing cannot.

ML routing costs: requires training data (cold-start disadvantage), prediction server adds latency, model drift risk.

9. Verified Performance Numbers

• The 46× TTFT improvement comes from agentic trace testing (prefix-heavy characteristics amplify KV reuse gains)
• VAST's 20× and Mooncake's 46× use different baselines; they are not contradictory
• llm-d's 40% baseline is an already-optimized vLLM deployment

10. Architectural Reasoning Conclusions

Short Term (1–2 years): Pure Software Products

LMCache/Mooncake/Dynamo software combinations cover tiered storage and routing. The six cluster challenges are addressed by software as best-effort: topology awareness via schedulers, consistency via version control, failure via degradation, bandwidth via cost models, versioning via model_id tagging, trust boundaries via software logical isolation.

AFD lands on Vera Rubin, changing decode bandwidth economics—more GPU HBM bandwidth goes to KV Cache, but KV Cache footprint simultaneously grows. AFD optimizes the compute side; KV Cache infrastructure optimizes the storage/transfer side; the two are complementary.

At this stage, KV Cache products are pure software—running on existing hardware's memory hierarchy (HBM → DRAM → NVMe → network).

Medium Term (3–5 years): NVLink-Attached Memory Nodes Emerge

Trigger conditions: 10,000-GPU-class inference clusters + agentic workloads going mainstream + software metadata overhead and consistency latency becoming measurable bottlenecks.

NVIDIA is the most likely player to take this path—controlling the NVLink standard + GPU design + NVLink Switch. The NVLink Memory Node doesn't need a GPU; it only needs large amounts of high-bandwidth memory + NVLink ports + basic KV block management. GPUs read directly via NVLink at ~50–100ns latency, with bandwidth in the same order of magnitude as GPU-to-GPU interconnect.

The software control plane doesn't need major changes (still tiered storage + routing), but the 100× latency gap between HBM and RDMA is filled by the NVLink Memory Node. Consistency within the NVLink domain is guaranteed by hardware; across domains, managed by software.

Long Term (5+ years): KV-Aware Hardware Matures

Block-level content addressing, write-once coherence, and session-scoped partition enter hardware standards. KV Cache "products" devolve into control planes—making cross-tier cost-optimized placement decisions (when to place in HBM, when to push to NVLink Memory, when to offload to RDMA, when to recompute). The data plane is taken over by hardware.

The most compelling proof will come from KV Cache crossing the training-to-inference boundary—if a KV generated during training can be saved in a checkpoint and directly deployed for inference use, KV Cache will have truly transcended from an engine's temporary state to a persistent asset. There is currently no product-level implementation of this path, but it defines the ceiling of "infrastructure-ization."

Final judgment: KV Cache infrastructure will go the way of NUMA—software defines policy, hardware provides mechanism. But in 2026–2028, it remains a software product. Hardware co-design must wait until inference cluster scale and agentic workloads push software solutions to their physical limits.

Checkpoints

1. Will Mooncake Conductor's scheduling logic be absorbed by other projects? Watch through end of 2026—whether Conductor's prefill+decode pairing and KV pool placement optimization appear in Dynamo or llm-d.

2. Will LMCache introduce its own scheduling component? Currently it's "independent storage + engine-native scheduling." If they start building scheduling, it signals the independent camp believes routing must leave the engine.

3. NIXL plugin ecosystem expansion speed. By end of 2026, if 2+ of VAST Data/NetApp/Pure Storage become native plugins, NVIDIA's "storage vendor integration" strategy is replicating rapidly.

4. AFD's real-world production performance. After Vera Rubin NVL72 + LPX ships at scale in 2026 Q3–Q4, the hidden state transfer overhead between attention (GPU) and FFN (LPU) becomes a measurable metric. If transfer overhead doesn't become a critical bottleneck, AFD's value in improving KV Cache effective bandwidth is validated.

---

Sources & Disclaimer: This article is based on publicly available information, drawing from vLLM official documentation and RFC #38260/#44223, Mooncake paper (FAST 2025 Best Paper) and GitHub documentation, LMCache blog and academic paper (arXiv 2510.09665v2), NVIDIA Dynamo/NIXL/Groq 3 LPX/Vera Rubin official documentation and technical blogs, DDN and VAST Data official blogs, GKE Inference Gateway documentation, llm-d project documentation and CNCF announcements, Ray 2.56.0 release notes, Samsung CXL CMM-D white paper, Astera Labs Leo Memory technical documentation, CXL-SpecKV (arXiv 2512.11920), Spheron Network deployment guide, and SiliconANGLE RAISE Summit 2026 coverage. This article does not constitute investment advice. Data herein is current as of July 12, 2026.