OCP 2026: KV-Cache Pooling Enters Validation — Returns Depend on Reuse and Transfer Cost

OCP 2026: KV-Cache Pooling Enters Validation

Returns depend on reuse and transfer cost

OCP 2026 pushes context-cache work onto three more concrete questions: can capacity be allocated where demand sits, can cached state be retrieved in time, and can the gains be measured consistently. Samsung demonstrated elastic memory allocation across instances via semantic emulation of a Dynamic Capacity Device (DCD Emulation); XCENA showed prefetching and hot-prefix pinning; Micron proposed a capacity-aware reuse model. These are, respectively, a scheme demonstration, software co-design, and early model validation — a unified KV service standard has not yet formed. [1][2]

Compared with simply adding memory capacity, the more notable shift this year is that software has started to manage where cached state goes and when it is used. For a token factory, how much value external memory or SSDs can deliver is decided by the recomputation they actually save, the retrieval cost, and the service objective together. To read these releases correctly, start by separating the three classes of cache demand they serve.

First separate active sessions, parked sessions, and shared prefixes

While a model generates subsequent tokens, it repeatedly reuses the keys and values computed from earlier context — commonly called the KV cache. It is inference intermediate state, distinct in purpose from the user's original text, vector databases, or an agent's long-term knowledge. Stored well, it avoids redundant computation; stored in the wrong place, it can trade a wait for computation for a wait for data movement.

The first class is active KV from ongoing generation, potentially accessed at every output step and most sensitive to bandwidth and latency. The second class is historical KV left by parked sessions — for example, an agent's context while it waits on a tool call or user approval — whose value depends on how soon the session returns. The third class is prefixes shared across requests, such as reused system prompts, whose value depends on how many requests actually reuse them.

These three demands cannot be compared by byte count alone. The same-sized cache: an active session needs continuous reads; a parked session needs to be ready the moment it resumes; a shared prefix may repeatedly save expensive input computation. A prefix shared by many sessions occupies one physical copy; conversely, identical text appearing after different preceding context cannot be reused by default, because its representation already carries the influence of that context. Capacity formulas and model differences are in the technical appendix.

The first advance lets capacity flow to where instances need it

The dynamic-capacity mechanism has a protocol basis. The original CXL 3.0 specification already includes Dynamic Capacity Device mechanisms; this year's increment is connecting that capacity-adjustment idea to concrete inference software. [11] Samsung's Pangaea v3 orchestrates the memory pool through Kubernetes, and LMCache bridges inference caching; together they demonstrated a path where host-visible capacity follows instance demand.

Under fixed allocation, one instance may keep evicting long context while another's capacity sits idle. Raising total pool capacity relieves pressure but can still leave idle slices that busy instances cannot use. Elastic allocation tries to hand that resource to the instance that needs it, letting capacity management follow the workload. For multi-model, multi-tenant platforms, the question moves one step further, from "how much memory per server" to "which requests should occupy which stretch of capacity, and when is it reclaimed."

Samsung's OCP site reports that in this agentic-workload demonstration, elastic allocation cut time-to-first-token (TTFT) by up to 38% versus static allocation, and goodput — throughput meeting the service objective — reached up to 3.3× the static baseline. [3] The result shows the potential of reducing idle capacity and recomputation under this specific workload; the joint talk still explicitly labels it DCD semantic emulation. The other set of DRAM-baseline results and the experimental hardware are listed in the technical appendix and must not be interchanged with these multiples.

Moving from demo to service still requires answering what reclamation costs. Once an instance has taken idle capacity, if the original tenant suddenly returns, should the platform migrate the cache, wait for release, or let the request recompute? Until an adjustment completes, can the scheduler know precisely whether capacity and data are ready? These are not capabilities this result already guarantees; they are new acceptance questions that elastic pooling adds. Dynamic allocation removes the waste of static slicing and brings resource contention and control latency into the product boundary.

The second advance gets cached state ready before it is used

Timely retrieval must solve two problems at once: knowing in advance what will be read, and moving it at the right granularity. Prefetching lets the system start reading before a request actually consumes the cache; hot-prefix pinning keeps frequently shared prefixes in the fast tier, avoiding repeated read-backs. XCENA's Maru integrates with LMCache and showed how both hints are used along a hybrid CXL-and-SSD path, letting device DRAM absorb the imminent accesses. [2]

Access performance close to DRAM here comes from data being ready in advance and hits landing in the fast tier — it must not be read as NAND itself acquiring DRAM latency. For a product team, an interface that admits access intent, prefetch deadlines, and pinning policy says more about what the software must do than a standalone emphasis on a unified address space. When access is unpredictable, the system should still allow misses and recompute in time.

Another Samsung demonstration this year wired Flexible Data Placement (FDP) and a larger NVMe maximum transfer size (MDTS) into LMCache, targeting write amplification and host-CPU overhead respectively. [3] They let data layout and I/O granularity follow the cache lifecycle. The actual endurance and CPU gains still depend on firmware, placement policy, request size, and reclaim behavior — they cannot be derived from a single drive's peak bandwidth.

The two paths sit at different points of the software stack. FDP and MDTS mainly change SSD data placement and I/O organization; prefetch and pinning require the upper scheduler to pass future access intent down. Platform teams therefore cannot compare storage by capacity price alone: with the same hardware, whether the software knows which blocks are about to be read and which shared prefixes deserve residency may decide whether the cache layer delivers inference gains. For accesses the application cannot predict accurately, a normal miss-and-recompute path must remain.

The third advance makes cache benefit consistently measurable

The modeling framework Micron proposed this year connects the working set produced by the workload, per-tier capacity, and retrieval cost. The core concept is effective reuse: a piece of context that will in theory be accessed again only realizes that reuse if it is still held somewhere retrievable in time when the access arrives. Insufficient capacity triggers eviction and recomputation, so the benefit of adding capacity can appear abruptly once the working-set threshold is crossed, and further additions may not help equally. [1]

This gives the first two demonstrations a shared evaluation question. Whether dynamic allocation actually gives capacity to instances that reuse it, whether prefetch completes before its deadline, and whether pinning crowds out more valuable data must all be answered against real request distributions. Micron has completed initial dense-model validation on GH200; more memory tiers and MoE remain open. The framework can help locate capacity inflection points; it cannot replace replaying actual traffic.

Acceptance metrics should count valid requests meeting TTFT, inter-token, and end-to-end deadlines, observed together with quality, error rate, and recovery behavior. A high overall hit rate may still yield little if the hits are cheap short prefixes; conversely, reusing a few expensive long prefixes can release substantial GPU time. So record token-weighted hits, saved prefill time, backfill queue tail latency, per-session resident byte-seconds, and device write volume — and finally compute the total cost of one qualified session.

Shared interfaces and evaluation methods are still forming

For different products to become interchangeable, they still need common operational semantics. On slide 19 of the joint talk, XCENA proposed that OCP's Composable Memory Systems (CMS) community could discuss and standardize a policy API, motivated precisely by giving schedulers and devices a shared understanding of operations like prefetch and pinning. The current evidence supports a standardization proposal; it does not license the claim that a unified interface has been approved. [2]

Evaluation methodology has its own gaps. On slide 16, Micron lists what is still missing: vendor-neutral workload descriptions, capacity-aware reuse metrics, a method for comparing memory tiers, and result formats — and suggests members align scope, draft v0.1 together, have partners validate, and review jointly. What is complete is the analytical framework and initial validation; multi-vendor, multi-tier joint testing is still ahead. [1]

The two agendas complement each other. Operational interfaces decide whether systems can coordinate; evaluation methods decide whether the coordination actually works. Even if two products both expose a prefetch operation, if one returns "task accepted" and the other "data consumable," upper layers cannot swap them; even if both report hit rates, comparing per-block, per-token, and successful-requests-only counting can distort results. These are semantics to be clarified in future interfaces and tests — not promises any vendor has made.

For buyers, "the standard is under discussion" does not mean waiting is the only option. You can pin engine, connector, and backend versions now, validate local gains with replayable traces, and write prefetch completion, capacity rejection, deletion, and failure fallback into your own acceptance requirements. But adopting a runnable implementation and being able to promise seamless vendor replacement in the future are still separated by interface stability and interoperability testing. Product contracts should not book that potential replaceability as delivered capability.

Adoption should start from real session traces

The first step is to build three contrasts from real session traces — no reuse, local reuse, external cache — replaying long and short requests, tenant bursts, and failure recovery under the same GPU budget and SLO. Only when saved recomputation and released expensive capacity durably cover storage, transfer, software operations, and risk does another cache layer become a good investment. What a token factory ultimately needs is a system that knows which computation is worth remembering, for how long, and how to recompute reliably when retrieval arrives too late.

The OCP 2026 signal is not that all KV schemes have matured to the same stage. Elastic pooling has begun validating capacity that follows demand; the storage path has begun understanding cache lifecycles; a common language for interfaces and evaluation has entered discussion. What is worth watching next is whether these demonstrations harden into systems with stable versions, clear failure semantics, and multi-vendor validation. For operators, the core of the investment case remains: how much computation is actually saved under a stated response objective, and whether the added state and failure responsibility can be carried indefinitely.

Technical appendix

The following consolidates experimental setups, capacity and latency derivations, data paths, and service-correctness requirements, so the schemes from this year can be placed into one validation process.

1 Experimental configurations and comparison baselines

Slides 10 and 12 of the Samsung joint talk explicitly label dynamic pooling as DCD Emulation — reproducing dynamic-capacity semantics on existing hardware as a rehearsal for native CXL 3.x DCD platforms. Slide 11 separately reports, against a 96GB DRAM baseline, a 52% TTFT reduction and 7.1× goodput, with the gate set at TTFT no more than 60 seconds. Those numbers use a different baseline than the site's 38% and 3.3× versus static allocation; they cannot be spliced into one conclusion. [2]

Slide 7 lists the experimental configuration as MD220 modules and an RTX PRO 6000 with 96GB of GDDR7. The MD310 product shown at the booth must not be written into this experiment's hardware; and the HBM in generic tiering diagrams cannot substitute for the concrete platform's GDDR7. [2]

Micron's initial validation used a GH200 NVL2, Llama 3.1 70B FP16, tensor parallelism TP=2, and configurations including 30K input tokens, 128 output tokens, and a fully reusable prefix. It validates trends for dense models and this class of working set first; MoE, newer KV optimizations, and more memory tiers are listed as future work. [1]

2 How capacity formulas and backfill times are computed

For an ordinary dense-attention model, uncompressed KV volume can be estimated as 2 × layers × KV heads × head dimension × tokens × bytes per element, then adjusted for concurrency, blocking waste, and replicas. You must use the number of KV heads, not a generic head count; grouped-query attention, sliding windows, and latent attention all change the capacity model. Models with equal parameter counts cannot share one cache-capacity table.

For an already-hit block, retrieval time should include directory lookup, queuing, the actual transfer, decompression or format conversion, and the time to restore it to an engine-consumable state. A hit only pays if its cost on the critical path is below recomputing that prefix. Effective bandwidth should be the measured bottleneck of the whole path, not an SSD's sequential-read spec sheet substituted into GPU recovery time. If transfer and compute can overlap, compare only the unhidden portion; if many requests recover simultaneously, also count queuing on the shared link.

Deciding whether to keep a block further multiplies by the probability and count of future reuse, then subtracts write-out cost, residency cost, and the opportunity cost of evicting other valuable cache. A session hours long that never returns may be worth less than a short, high-frequency system prefix. LRU is a starting point; production policy should learn per-business revisit intervals, reusable token counts, and recompute cost, and apply write admission so one-shot context is not dumped wholesale into external storage.

As a purely illustrative example: a 2GiB cache block on a path with 20GiB/s effective bandwidth needs about 100ms for the data transfer alone. If lookup, queuing, and conversion add 150ms while recomputation takes 180ms, the "hit" is slower; if recomputation takes two seconds, retrieval clearly pays. There is no universal long-context threshold here: a smaller model, more prefill compute, or a more congested shared link each move the boundary. The scheduler must allow abandoning a hit on a remote block and treat timely recomputation as a normal path.

One easily confused boundary: the KV handoff of prefill/decode disaggregation passes necessary state between two execution phases of the same request; cross-request reuse saves prefill that would otherwise be redone for future requests. The former happens even with zero repeated traffic and cannot be replaced by cache-hit metrics — handoff bandwidth and latency need their own evaluation. vLLM's documentation frames prefill/decode disaggregation around independently tuning TTFT and inter-token latency while controlling tail interference; it does not promise a throughput gain from splitting alone. [6]

3 Deployment boundaries of data paths and storage lifetime

GPU HBM suits the active decode working set, but its capacity also competes with model weights, activations, and runtime buffers. Host DRAM can absorb brief pauses and hot prefixes, at a cost set by cross-processor memory placement (NUMA), pinned pages for stable transfer, the CPU-to-GPU link, and copy engines. Platforms with high-bandwidth coherent interconnect between CPU and GPU cannot reuse the same recovery curve as ordinary PCIe servers. Micron's five-tier model offers a taxonomy from near memory, main memory, and extended memory down to context storage, but the bandwidth and residency ranges in its diagrams are architectural descriptions, not service levels every product guarantees. [4]

CXL pools can widen host-visible capacity and make allocation more elastic, but they do not automatically turn remote memory into HBM. CPU root ports, switch tiers, memory interleaving, OS mappings, and how the GPU actually reads this data must each be confirmed. A concrete counterexample is LMCache's DAX backend: the documentation explicitly describes copying data back into CPU memory objects on read, and its MP implementation does not yet guarantee index recovery after restart. "Addressable," "GPU-directly-consumable," "survives process exit," and "persists across power loss" are four different capabilities; procurement and APIs should not merge them. [5]

Two kinds of offload must also be distinguished: saving idle cache to a lower tier for later restoration, versus having active decoding fetch attention data from the lower tier at every step. The former mainly affects recovery time; the latter can bind every output token to a slower link. If transfers cannot be hidden behind compute, no external capacity guarantees streaming output speed. Near-memory compute or compression can move this boundary, but each needs separate validation of quality, compute capacity, and software compatibility — not a default dividend of ordinary capacity expansion.

Shared pools have boundaries too. Two hosts allocating from one pool do not thereby have correct cross-host sharing semantics; even with network KV copies eliminated, the GPU side still pays access, transfer, or compute costs. Pooling saves idle capacity and duplicate holding; uplink bandwidth, controller queuing, and failure domains may become the new central bottlenecks.

Capacity pools also need headroom. If every device byte is sold, transient backfill, migration, and replica rebuilds have nowhere to land. Cache-aware routing has its own failure mode: herding requests onto one node for hits worsens GPU or NIC queuing, while routing to idle nodes pays transfer. A good routing decision should jointly estimate the target's compute queue, cache readiness, and remaining output length — not simply pick the node with the most cached tokens. This, too, is new systems responsibility once caching graduates from library function to infrastructure service.

SSDs fit longer-lived, larger reusable state, but "cheap and roomy" alone does not produce inference gains. Cache block size, I/O coalescing, queue depth, prefetch lead time, and backfill bandwidth together determine recovery time. Too-small requests waste CPU and protocol overhead; too-large transfers can block small requests that urgently need to recover. A shared SSD pool also absorbs cross-tenant recovery bursts and cannot be sized from single-drive peaks.

SSD lifetime should be assessed from how many KV bytes are admitted per day, plus device write amplification, rewrites, and replica cost, then compared with the specific model's endurance rating. More cold data does not mean less write traffic; FDP's real gains still need firmware, placement policy, and reclaim behavior to cooperate. The SSD tier also takes shared-recovery bursts; single-drive sequential peaks do not represent the service's sustained capability.

4 Correctness of cross-instance reuse and tenant isolation

Cross-instance reuse must first answer whether this KV belongs to the current computation. Cache identity should bind model-weight version, token sequence and prefix, adapters, and the tokenization, positional encoding, attention, and quantization settings that change representations; the layout must stay compatible with the engine and parallel sharding. Version upgrades should switch namespaces rather than keep hitting on a human-readable model name. vLLM already folds parent block, tokens, LoRA, multimodal inputs, and an isolation salt into its prefix hash — cache identity was always more than text content. [7]

The engineering-sound pattern is to write immutable blocks completely first, then publish "readable" metadata; hold leases or reference counts during reads to prevent reclamation; treat failures as misses; and give recomputation a budget so a pool fault cannot trigger a cluster-wide recompute storm. Tenant isolation must cover authentication, directory visibility, quotas, deletion, and audit. The salt divides reuse scope and does not replace access control; at-rest encryption does not replace lifecycle management. If KV carries derived information from sensitive input, then when a user deletes a session there must be an explicit commitment on whether copies and indexes are invalidated together.

Beyond consistency, define post-invalidation behavior. If the directory claims a hit but the data was reclaimed, object generations must be detectable so a read cannot land on different content at a reused address; when a shard spans multiple GPUs, all required shards must be confirmed available before consumption begins. Drills for model upgrades, tenant deletion, and pool-node failure matter at least as much as normal hit testing. Cache may be disposable derived state — the service must not lose the ability to complete requests when the cache is lost.

5 Interface status and the session billing contract

For a CTO, the sensible boundary is to integrate engine connectors, mature transports, and storage backends while owning the business-specific admission, routing, priorities, capacity pricing, and SLO control. Product APIs should offer namespaced lookup, prefetch, pin, release, and delete, and return recoverable states such as capacity-rejected, timeout, and miss; "prefetch accepted" must not be wrapped as "data ready." LMCache already provides cache operations and quota interfaces, but soft quotas and asynchronous semantics still have to be handled correctly by the upper service. [8]

Session billing should not copy disk-capacity billing either. Occupying GPU memory while active and external cache while waiting on a tool are different resource costs; holding all waiting time at the most expensive memory tier masks the product's own inefficiency. More workable is to contract recovery deadlines and retention windows per service tier, then choose media internally. Enterprises can buy longer recoverable-session windows, but a cache-hit promise is not a data-persistence promise; business records still belong in an independent durable system.

Sources and caveats

The core on-site evidence for this article is Micron's "Tiered Memory for LLM Inference" and the joint Samsung, XCENA, Tensormesh, and Marvell talk, all public OCP 2026 materials. The standardization agenda appears on slide 16 of the former and slide 19 of the latter; neither is a published unified KV specification.

The Samsung site demo, DCD emulation, XCENA's hybrid path, and Micron's analytical model each retain their own platform and baseline; results are not stitched into one performance conclusion. Micron's September 2026 tiering article and the current vLLM and LMCache documentation serve only as architecture and implementation background; software semantics were read on October 17, 2026. The 2GiB/20GiB/s figures are an illustrative example; the service-interface and procurement suggestions are the author's analysis.

References

[1] Tiered Memory for LLM Inference — A Modeling Framework for KV-Cache Offload, Khayam Anjam | Micron / OCP Global Summit 2026 | 2026-10.
https://drive.google.com/file/d/1ML8XjdPeXdkdUAUNmOPIInQwKR718cRq/view

[2] Disaggregated CXL Memory Architecture for Agentic AI: Dynamic Pooling, Sharing, and PNM Offloading, Samsung, XCENA, Tensormesh, Marvell / OCP 2026 | 2026-10.
https://drive.google.com/file/d/1iK-aymMCRgd_qHZaWzRTVz1NENAW5yK9/view

[3] OCP Global Summit 2026, Samsung Semiconductor | 2026-10.
https://semiconductor.samsung.com/events/ocp-2026/

[4] Unlocking the Token Economy: Micron Powers AI's KV Cache, Micron | 2026-09-04.
https://www.micron.com/about/blog/memory/dram/unlocking-the-token-economy-micron-powers-ais-kv-cache

[5] Device-DAX (/dev/dax), LMCache | live documentation, retrieved 2026-10-17.
https://docs.lmcache.ai/kv_cache/storage_backends/dax.html

[6] Disaggregated Prefilling (experimental), vLLM | live documentation, retrieved 2026-10-17.
https://docs.vllm.ai/en/latest/features/disagg_prefill/

[7] Automatic Prefix Caching, vLLM | live documentation, retrieved 2026-10-17.
https://docs.vllm.ai/en/latest/design/prefix_caching/

[8] HTTP API, LMCache | live documentation, retrieved 2026-10-17.
https://docs.lmcache.ai/mp/http_api.html

[9] Architecture Overview, LMCache | live documentation, retrieved 2026-10-17.
https://docs.lmcache.ai/developer_guide/architecture.html

[10] Optimizing KV Cache Offloading to CMM-D in a CXL Switch-based Memory Pool, Samsung | 2026-06.
https://download.semiconductor.samsung.com/resources/white-paper/Optimizing_KV_Cache_Offloading_to_CMM-D_in_a_CXL_Switch-based_Memory_Pool.pdf

[11] Compute Express Link Specification r3.0 v1.0, CXL Consortium | 2022-08-01.
https://computeexpresslink.org/wp-content/uploads/2024/02/CXL-3.0-Specification.pdf