KV Cache Scheduling Engineering: After Compression to 7%

KV Cache Scheduling Engineering: After Compression to 7%

V4 Flash CSA+HCA compresses KV cache to just 2 MB for 100 concurrent requests. The scheduling problem shifts from cant fit to move fast and save prefill.

Inference Economics Trilogy · DeepSeek V4 Flash outputs at ¥2/M tokens, Claude Sonnet 5 at ¥107/M. The 53× gap isn't subsidy—it's physics. This series reverse-engineers a token's physical cost from pricing through architecture to scheduling.

Compiled: 2026-08-08 Positioning: Engineering deep-dive of the LLM Memory Architecture Evolution series Dependencies: memory/llm-memory-architecture-evolution.md + inference pricing physics floor analysis Anchor Models: DeepSeek V4 Flash / V4 Pro (CSA+HCA Hybrid Attention, 2026-04 technical report)

Introduction: One Table Shows What V4 Changed

In the V3 era, the core contradiction of KV cache management was "doesn't fit"—the 671B-parameter MoE model already consumed most of HBM, and the remaining space had to be carefully budgeted. V4's CSA+HCA hybrid attention compressed KV cache to 7% (Flash) and 10% (Pro) of V3.2, an order-of-magnitude change that fundamentally transformed the nature of the scheduling problem.

First, the parameter table:

One striking number: running 100 concurrent requests on V4 Flash, each averaging 3,200 tokens of context, the total KV cache is only 2 MB. This is not a typo—2 MB, not even a rounding error against HBM.

This means the center of gravity for KV cache hot-cold scheduling has shifted from "doesn't fit" to "moves fast enough." The problem hasn't disappeared, but it has transformed.

I. Problem Redefinition

KV cache management is not simply "offload when it doesn't fit." It is an online scheduling problem with multiple constraints. But in the V4 era, the constraints are fundamentally different from V3.

Three simultaneous physical processes:

1. Decode Step: Each token generated requires reading the entire KV cache (attention computation) + reading active weights
2. KV Growth: Each token generated adds a few bytes to KV cache—but on V4 Flash, only ~6.3 bytes/token, totaling ~25 KB for 4K context
3. Request Lifecycle: Request arrives → prefill (KV cache generated in one shot) → decode (KV cache grows incrementally) → complete (KV cache can be discarded or cached for reuse)

Scheduling Constraints: V4 Era vs V3 Era:

The core problem has transformed: Given N concurrent requests, KV cache capacity is no longer a constraint on V4. The real questions are: ① How to maximize prefix sharing to reduce redundant prefill computation (KV cache is nearly free, but prefill compute cost hasn't decreased); ② How CSA's sparse selection patterns affect KV cache reads (most KV cache entries are never read during attention computation); ③ How to schedule KV cache transfers between GPUs in a PD-decoupled architecture.

II. What Determines Hot vs Cold? Attention's Access Patterns

The principles in this section hold for both V3 and V4—attention access patterns determine hotness, regardless of model architecture. But V4's CSA+HCA architecture performs hot-cold tiering at the hardware level, which is a fundamental difference (detailed in Chapter VI).

CPU cache hotness is determined by temporal and spatial locality. KV cache hotness is determined by attention access patterns—a fundamental difference.

Finding 1: Attention Sink

Liu et al. (2023) discovered that transformer attention has an "anchor effect": the first few tokens and the last few tokens receive disproportionately high attention scores, regardless of what the subsequent query is.

This means there are natural hot spots in KV cache:

• First ~64 tokens (attention sink): Almost all subsequent queries attend to these → always hot
• Recent window (e.g., last 128 tokens): Recency bias, high probability of being attended to → hot
• Middle segment: Attention is mostly dispersed and sparse → cold candidate

Engineering implication: the middle segment of KV cache is a safe zone for offloading. As long as sink tokens and the recent window are retained, the accuracy loss in attention computation is usually acceptable.

Finding 2: Prefix Sharing

In production environments, a large proportion of requests share prefixes:

• System prompts (shared by all requests, several thousand tokens)
• Few-shot examples (shared by requests from the same application)
• Document/code context (shared across turns in the same session)

DeepSeek's measured data (DualPath paper): in API serving, approximately 30–40% of prefill computation can be eliminated through prefix reuse. In Agent multi-turn scenarios, this ratio reaches 98.7%.

This means: shared prefix KV cache is the highest-value hot data—computed once, reused many times. On V4, while the KV cache itself is tiny, the FLOP cost of prefill hasn't decreased (13B active parameters × 2 × prefix_length). The value of prefix sharing has therefore relatively increased.

Finding 3: Semantic Importance Fluctuation

Not all middle tokens are cold. Certain tokens become important under specific queries:

• Entity names, numbers, keywords
• Field labels in structured data
• Function signatures in code

But semantic importance is query-dependent—cannot be predetermined which tokens will become hot. This is a fundamental difference from CPU caches: the determining factor for hotness is not historical access patterns, but future query content.

Hot-Cold Tiering Model (V4 Perspective)

Synthesizing the three findings, KV cache hotness can be modeled as:

On V4 Flash, the full KV cache for 4K context is only ~25 KB. Even if P3 cold segment is 60%, that's only ~15 KB—offloading it to DRAM saves space not worth the page-in latency overhead. In the V4 era, the significance of hot-cold tiering shifts from "saving space" to "reducing bandwidth contention" and "guiding prefix reuse."

III. Design Space of Existing Systems

vLLM PagedAttention: Virtual Memory Model

Core Design: Borrowing from OS virtual memory, KV cache is split into fixed-size blocks (typically 16 tokens/block), with a block table managing logical-to-physical mapping.

• Advantages: Eliminates fragmentation, block-level allocation/reclamation, supports non-contiguous physical storage
• Hot-Cold Handling: P0/P1 in HBM; P3 can be evicted—but original PagedAttention has no DRAM/SSD tiers; eviction means discarding
• Prefix Sharing: Implemented through RadixAttention extension (later SGLang's contribution)
• Limitation: Fixed block granularity (16 tokens); for hot spots like attention sink (only a few tokens), there's internal fragmentation
• V4 Adaptation: On V4 Flash, a 16-token block is only ~101 bytes; PagedAttention's block table metadata may be larger than the data itself. Need to increase block size to 256–512 tokens, or align with CSA blocks

SGLang RadixAttention: Prefix Tree Reuse

Core Design: Organizes all active requests' KV cache into a Radix Tree by prefix. Requests with identical prefixes share the same KV blocks.

• Advantages: Automatically discovers prefix sharing without manual system prompt annotation
• Hot-Cold Handling: Prefix tree nodes managed by reference count. High-ref-count nodes stay in HBM; low-ref-count nodes can be downgraded
• Limitation: Prefix matching is exact—"nearly identical but not quite" prefixes cannot be reused
• V4 Adaptation: On V4, RadixAttention's value is not in saving HBM space (KV cache is already tiny) but in saving prefill compute. Each shared prefix node means a group of requests doesn't need to re-run prefill

DeepSeek's Practice: Three-Tier Storage + Prefix Caching (V4 Production Environment)

DeepSeek V4's production environment already runs a complete three-tier storage architecture. The DualPath paper (2026-02) confirms this design on a 660B production model.

Tier Architecture:

HBM (80GB)     ← Active requests' KV cache + P0/P1 hot spots (V4 Flash KV only ~2 MB)
  ↕ PCIe Gen5 (64 GB/s)
DRAM (1–2TB)   ← Prefix cache pool + cross-request KV reuse
  ↕ NVMe Gen5 (7–14 GB/s)
SSD (4–8TB)    ← Cross-session persistent cache + Agent long-chain history

Design Changes in the V4 Era:

1. HBM Tier's Role Changed: In the V3 era, HBM was scarce—KV cache and weights competed for the same space. V4 Flash's KV cache is only ~2 MB; HBM's KV budget is virtually unlimited—the first tier alone suffices
2. DRAM/SSD Tier's New Role: No longer "KV cache overflow zone" but "prefix reuse pool." The value is not in saving HBM but in reducing prefill computation
3. PD-Decoupled Architecture: Prefill nodes (P nodes) and Decode nodes (D nodes) are separated; KV cache must be transferred over the network from P to D. V4's tiny KV cache makes this transfer nearly free—6.3 bytes/token × 3,200 tokens = 20 KB/request; 100 requests = only 2 MB; at PCIe 64 GB/s, 0.03ms to transfer
4. Asynchronous page-in still exists, but the main reason shifts from "HBM can't fit" to "KV cache needs cross-node transfer in PD-decoupled architecture"

IV. Scheduling Strategy Design

Problem Formalization

Scheduling decisions at each decode step:

Inputs:

• Current HBM remaining space S_hbm (usually not a constraint on V4 Flash)
• KV cache state of all active requests (which tier they're on)
• Each request's attention pattern history (CSA layer sparse selection records)
• Prefix tree state (how many requests share each prefix)

Outputs:

• List of blocks to page-in to HBM this step (on V4, mainly in PD-decoupled architecture)
• List of blocks to evict from HBM to DRAM this step (rare on V4)
• Prefix cache update strategy

Objectives (V4 Era Priority Reordering):

1. Maximize prefix cache hit rate (reduce prefill compute—top priority)
2. Minimize KV cache transfer latency in PD-decoupled architecture (cross-node bandwidth contention)
3. Minimize decode latency (CSA layer sparse KV read pattern optimization)
4. Minimize HBM footprint (no longer a constraint)

Strategy 1: Tiered LRU + Attention Sink Exemption

The most basic strategy. Each tier maintains an LRU queue, but P0 (attention sink) is never evicted.

• HBM Queue: P0 (permanent) → P1 (recent window sliding with decode) → P2 (by LRU)
• DRAM Queue: P2/P3 evicted from HBM, by LRU
• SSD Queue: Evicted from DRAM, by reference count (prefix reuse value)

V4 Adaptation: On V4 Flash, LRU's role is greatly reduced. HBM has ~67 GB available (after subtracting 13B active weights' ~13 GB), while total KV cache is only ~2 MB. The LRU queue never fills—eviction never triggers. This strategy degenerates into "all KV in HBM."

Advantages: Simple, predictable, low overhead. Disadvantages: LRU assumes "recently accessed will be accessed again"—not entirely true for KV cache. But on V4, this drawback is irrelevant since there's no eviction pressure.

Strategy 2: Attention Score-Driven Prefetch

Leveraging information from attention computation itself.

Core Observation: At each decode step, attention scores already tell you which historical tokens are being attended to. Blocks with high attention scores can be marked "hot," and low ones "cold."

V4 Adaptation: On V4, the CSA layer does something similar at the architecture level—only retaining top-k compressed KV entries. Strategy 2's role shifts from "scheduling optimization" to "software intervention on CSA's selection strategy"—if the inference engine can capture CSA's selection patterns, it can prefetch more precisely.

Advantages: Precisely reflects actual access patterns. Disadvantages:

• Additional compute overhead (statistics + decision-making)
• CSA's selection is determined by model weights, not runtime attention scores—limited intervention space
• HCA's dense attention doesn't need prefetch optimization

Strategy 3: Prefix Tree + Reference Counting

Specifically for P2 (shared prefix) management. This is the highest-ROI strategy in the V4 era.

Design:

1. Maintain a Radix Tree where each node represents a KV cache block sequence
2. Each node has a reference count ref_count (how many active requests are using this prefix)
3. Nodes with ref_count > 0 always stay in HBM (on V4, nearly all nodes fit—the space is not an issue)
4. Nodes with ref_count = 0 but with reuse history go to DRAM (prefix cache)
5. When DRAM is full, evict to SSD based on "last used time + historical hit rate" composite score

Key Optimization—Adaptive Prefix Granularity:

• Prefix tree branch granularity is not fixed. For highly shared prefixes like system prompts, use coarse granularity (hundreds of tokens per node) to reduce management overhead
• For multi-turn conversation context, use fine granularity (tens of tokens) to improve hit rates

Core Value on V4: V4 Flash's prefill compute is 2 × 13B = 26 GFLOP/token. A 1,500-token system prompt reused 99 times saves 1,500 × 26 GFLOP × 99 = 3.86 PFLOP of prefill FLOP—equivalent to saving ~0.24s of compute at 16 PFLOP/s. Meanwhile, storing these prefixes' KV cache costs only 1,500 × 6.3 bytes = 9.5 KB—storage cost is negligible, compute savings are enormous. Prefix Tree on V4 is purely a "compute saver."

Strategy 4: Speculative Prefetch

Core Idea: Don't wait for the decode step to discover a block is needed before paging it in—predict ahead.

Signal Sources:

1. Request prompt structure: In multi-turn dialogue, the next turn will most likely attend to the previous turn's response → preemptively pull the previous turn's KV cache from DRAM back to HBM
2. CSA selection pattern history: CSA's top-k selection has some pattern stability; if certain KV entries have been consistently selected over the past N steps, prefetch them
3. Intra-batch correlation: Similar requests within the same batch can inform each other's CSA selection patterns

Implementation: Execute page-in asynchronously during decode gaps (while GPU is doing matrix multiplication). A 16-token block of KV cache on V4 Flash ≈ 101 bytes. PCIe Gen5 at 64 GB/s can transfer 101 bytes in 0.001ms—completely hidden within the 10–15ms decode compute time.

Quantification: V4 Flash's blocks are so small that Speculative Prefetch's engineering value on V4 primarily manifests in PD-decoupled architecture—after Prefill nodes compute KV cache, they need to transfer KV to Decode nodes. Predicting which requests will be routed to which D node and transferring ahead of time is what truly eliminates latency.

V. Strategy Comparison Benchmarks: What the Data Tells Us

The four strategies described above are four quadrants of the design space. But the question engineers actually need to answer is: how has their relative value changed in the V4 era?

5.1 Public Data Points

Three sources of production-grade or paper-grade data:

DeepSeek DualPath (2026-02 paper, 660B production model, V4 production environment validation)

• Workload: Agentic multi-turn dialogue, average 157 interaction turns
• Average context length: 32,700 tokens
• New tokens per turn: only 429 (all else reusable)
• KV-Cache hit rate: 98.7% ((32700−429)/32700)
• Cache-Compute Ratio: 22 GB/PFLOP
• After dual-path loading (Storage-to-Decode + Storage-to-Prefill): throughput improvement 1.87×/1.96×
• Core bottleneck: not compute, but storage I/O bandwidth—PE-side NIC saturated, DE-side NIC idle
• V4 Adaptation Reading: DualPath's core finding is even more extreme on V4—KV cache is smaller, but prefill compute savings value is higher. Bottleneck shifts further from "storage I/O bandwidth" to "network bandwidth" (in PD-decoupled)

SGLang RadixAttention Paper and Production Measurements (various workloads)

• RadixAttention improves cache hit rates by 3–5× compared to no-prefix-cache baseline
• Speedup by scenario (measured by TTFT):

vLLM PagedAttention Paper

• Block-level management eliminates internal/external fragmentation; memory utilization improved from ~60% (contiguous allocation) to ~95%+
• But original PagedAttention has no DRAM/SSD tiers—eviction means discarding, no cross-tier scheduling
• Prefix cache was added later as an extension (vLLM Automatic Prefix Caching); hit rate depends on workload prefix repetition
• V4 Adaptation Note: PagedAttention's block-level management faces a granularity problem on V4—a 16-token block is only ~101 bytes; block table metadata overhead is disproportionately high. Block size needs redesign

5.2 Four-Strategy Horizontal Comparison (V4 Perspective)

Synthesizing the above data and engineering inference, the four scheduling strategies ranked by value in the V4 era:

5.3 Key Judgments

The data reveals three V4-era engineering conclusions:

Judgment 1: Prefix sharing value increases (not decreases) on V4. In the V3 era, prefix sharing saved HBM space + saved prefill compute—both values coexisted. In the V4 era, HBM space is no longer scarce, but prefill compute cost hasn't decreased—every prefill saved equals 26 GFLOP/token saved (V4 Flash). Prefix Tree shifts from "space optimizer" to "pure compute optimizer."

Judgment 2: Attention Score-driven scheduling value shrinks dramatically on V4. The CSA layer performs "sparse selection" at the architecture level—most KV entries are never read during attention computation. Software-level attention score statistics have limited incremental value on top of CSA. The only valuable scenario is analyzing CSA's selection patterns for prefetch decisions in PD-decoupled architecture.

Judgment 3: Speculative Prefetch shifts from "local page-in" to "cross-node KV transfer." V4 Flash's KV cache is too small (~2 MB); local HBM↔DRAM page-in has zero pressure. But in PD-decoupled architecture, P nodes need to transfer KV to D nodes after prefill—predicting routing and pre-transferring is the real battleground for Speculative Prefetch in the V4 era.

Recommended Combined Strategy (V4 Era): Strategy 3 (Prefix Tree) as the core—highest ROI optimization. Strategy 1 (LRU) degenerates to default fallback, no special investment needed. Strategy 2 (Attention Score) has research value only for CSA pattern analysis. Strategy 4 (Speculative Prefetch) is a key optimization in PD-decoupled deployments. DeepSeek's DualPath framework is essentially the combination of Strategy 3 + Strategy 4.

VI. Compound Impact of CSA+HCA on Scheduling Strategy

V4's CSA (Compressed Sparse Attention) and HCA (Heavy Compression Attention) don't just compress KV cache—they redefine the meaning of "hot vs cold" at the architecture level.

Four Generations of Attention Mechanism Comparison

Standard MHA assumption: The ~320 KB above corresponds to ~100B parameter model, 96 layers, 64 heads, 128 head_dim, FP16. | Per Request (4K context) | ~1.3 GB | ~121 MB | ~350 KB | ~25 KB | ~36 KB | | HBM Concurrent Requests (80GB minus weights) | 7–8 | 27–28 | ~10K | ~1.3M | ~900K | | 16-token Block Size | 5.1 KB | 496 bytes | 1.4 KB | 101 bytes | 144 bytes | | HBM→DRAM page-in latency/block | 0.08ms | 0.007ms | 0.02ms | 0.002ms | 0.002ms |

KV Cache Shrinking Cascade: Standard MHA → V3 MLA is ~10× compression (compressing latent dimensions). V3 MLA → V3.2 optimized is further ~340× compression (presumably more aggressive latent dimensions + sharing mechanisms). V3.2 → V4 Flash is another ~14× compression (CSA sparse selection + HCA block-level compression). From MHA to V4 Flash, the total compression ratio reaches ~50,000×.

CSA's Additional Impact on Scheduling: Architecture-Level Hot-Cold Tiering

CSA's core mechanism is: before attention computation, it uses a learnable compression network to perform top-k selection on the KV sequence, retaining only high-information KV entries for attention. This means:

Most KV cache entries are never read during attention computation. Not "rarely read"—but architecturally, deterministically skipped. CSA-layer KV cache naturally splits into two tiers:

• Selected entries (hot): CSA's top-k selection results, attended to in the current step
• Unselected entries (cold): Not attended to in the current step, but future queries may select them

This is far more aggressive than V3's approach of using LRU or attention scores for hot-cold tiering. V3's MLA has all KV entries participating in attention; hot-cold tiering is a software-level scheduling decision. V4's CSA determines at the architecture level which KV entries are read; the software only decides "where to put the unread entries."

Engineering Implication: On V4 Flash, CSA-layer cold KV entries can be aggressively offloaded—but in practice, no offload is needed because the entire KV cache (including cold entries) is only ~2 MB. The architecture-level hot-cold tiering is more about information value—the scheduler can leverage CSA's selection patterns to optimize prefix cache eviction: if certain KV entries haven't been selected by CSA in the past N steps, their reuse value is indeed low and they can be downgraded from the prefix tree.

HCA's Impact on Scheduling: Dense Attention Layers Cannot Be Offloaded

In contrast to CSA, HCA (Heavy Compression Attention) performs intra-block standard attention—all KV entries participate in computation. This means:

• HCA-layer KV cache cannot be offloaded: All entries may be attended to; there's no CSA-style architectural skip
• HCA-layer KV cache is "all hot": Regardless of query, all tokens in the block are read

CSA+HCA hybrid architecture therefore creates a natural dual-layer hot-cold structure:

This dual-layer structure is V4-exclusive. V3's MLA has all homogeneous layers; hot-cold tiering relies entirely on software. V4's CSA+HCA provides hot-cold labels at the hardware level; the software scheduler only needs to optimize prefix reuse and PD-decoupled transfers on top.

Engineering Trade-off: Rethinking Block Size

V4 Flash's per-token KV cache is only ~6.3 bytes. If沿用 V3-era 16-token blocks:

• Per block size: 16 × 6.3 = 101 bytes
• Block table entry (logical address + physical address + status flags): ~16–24 bytes
• Metadata overhead: 16–24%

This means nearly 1/4 of overhead for block management. V4 needs significantly larger block sizes:

Recommended V4 Flash default: 256-token block. Metadata overhead drops to ~1.2% while maintaining sufficient scheduling granularity.

VII. Engineering Challenges

Challenge 1: KV Cache Transfer in PD-Decoupled Architecture

In the V4 era, KV cache capacity is no longer a bottleneck, but Prefill-Decode separation architecture introduces a new scheduling dimension.

PD-decoupled core flow:

1. P node completes prefill, generates KV cache
2. KV cache transfers from P node to D node
3. D node begins decode

V4 Flash Transfer Cost:

• Per-request KV cache: 3,200 tokens × 6.3 bytes = 20.2 KB
• 100 concurrent requests: 2.02 MB
• PCIe Gen5 64 GB/s: transfer time 0.03ms (negligible)
• 10G network 10 GB/s: transfer time 0.2ms (still negligible)

This is a 5,000× improvement over V3 (100 concurrent × 3,200 × 31 KB = 9.9 GB; PCIe transfer 0.15s). PD-decoupled architecture on V4 has virtually zero latency penalty.

But the real bottleneck is network bandwidth contention. If MoE expert parameters are simultaneously transferring between GPUs, KV cache transfer competes with expert loading for the same PCIe/NVLink bandwidth. V4 Flash's 284B total parameters are distributed across 8 GPUs; each decode step needs to load routed experts—expert loading bandwidth can be thousands of times greater than KV cache transfer.

Challenge 2: Block Merging and Compression

When a request completes, its KV cache may be released or downgraded. But if another request shares part of its prefix, the block needs splitting: shared portion retained, unique portion released.

Radix Tree naturally supports this via node split/merge. On V4, the special consideration is that blocks are so small (256-token block = only 1.6 KB) that merge/split operation overhead is relatively high. It's recommended that V4's Radix Tree use larger node granularity (e.g., 512–1024 tokens) to reduce operation frequency.

Challenge 3: Cross-GPU KV Cache Distribution

During multi-GPU tensor parallelism, KV cache is scattered across different GPUs. V4 Flash's CSA+HCA layers have different distribution strategies across GPUs:

• CSA layer's top-k selection results require allreduce synchronization
• HCA layer's intra-block attention is computed in parts across different GPUs
• But KV cache is so small (per GPU possibly only ~0.25 MB) that cross-GPU synchronization overhead dominates over data transfer

Scheduling decisions need to coordinate across multiple GPUs:

• Same request's KV cache should be on the same GPU (avoiding cross-GPU reads)
• Prefix-shared blocks only need one copy on one GPU
• V4's New Challenge: CSA's top-k selection is determined by model weights, not runtime scheduling—the scheduler can only optimize prefix and PD decisions "below" CSA's choices

Challenge 4: Precision Loss

Offloading KV cache from HBM to DRAM/SSD itself causes no precision loss (bit-exact). V4 Flash's FP8 KV cache is already 8-bit quantized; further quantization to INT4 yields limited benefit:

• V4 Flash FP8: 6.3 bytes/token (current production state)
• Theoretical INT4: 3.15 bytes/token (3–5% precision loss)
• Absolute savings: 3.15 bytes/token × 3,200 × 100 = 1 MB

Saving 1 MB of HBM is not worth 3–5% precision loss. In the V4 era, KV cache quantization is essentially unnecessary—the cache is already too small. Tiered quantization (P0 FP8 / P3 INT4) also yields insignificant benefits, since the P3 segment's entire KV cache may be only ~1 MB.

VIII. End-to-End Case Study: 8×H100 Running V4 Flash MoE

The qualitative analysis in previous sections needs a "fully computed case" for validation. Let's walk through a concrete deployment scenario end to end.

8.1 Scenario Setup

Hardware: 8×H100 80GB Server

• Total HBM: 640 GB
• Per-GPU Bandwidth: 3.35 TB/s
• 8-GPU Aggregate Compute (FP8): ~16 PFLOP/s
• PCIe Gen5: 64 GB/s
• DRAM: 1 TB
• NVMe SSD: 8 TB

Model: DeepSeek V4 Flash MoE

• Total Parameters: 284B
• Active Parameters per Token: 13B
• Attention Mechanism: CSA+HCA Hybrid
• KV Cache per Token (FP8): ~6.3 bytes (7% of V3.2 MLA)
• Per-Token Inference FLOP: 2 × 13B = 26 GFLOP
• Native Context Length: 1M tokens

Weight Deployment: 284B FP8 ≈ 284 GB, distributed across 8 H100s. Each GPU handles ~35.5 GB of weights. Active parameters 13B spread across 8 GPUs ≈ 1.6 GB/GPU (shared layers + routed experts), but the active expert weights loaded per step are larger. Conservative estimate: ~13 GB active per GPU (including MoE routing all-to-all communication buffers); remaining ~67 GB/GPU available for KV cache and other buffers.

KV Cache Budget: 640 GB − 284 GB (weights) = 356 GB. But actual KV cache needed is only ~2 MB (see below); HBM space is absolutely not a constraint.

8.2 Workload

Shared Prefixes:

• System Prompt: 1,500 tokens (shared by all 100 requests)
• Few-shot Examples: 500 tokens (shared within each group of 20 requests; 5 groups total)

8.3 Strategy A: LRU + No Prefix Sharing

Each request independently computes full prefill.

Prefill Computation:

• Total prefill tokens: 20 × 8,000 + 80 × 2,000 = 320,000 tokens
• Per-token prefill FLOP: 2 × 13B = 26 GFLOP
• Total prefill FLOP: 320,000 × 26 GFLOP = 8.32 PFLOP
• At 16 PFLOP/s aggregate compute: 8.32 / 16 = 0.52s (theoretical)
• Actual with MoE expert loading overhead (284B total params distributed across GPUs; each step loads routed experts): approximately 0.7–0.9s

KV Cache Footprint:

• Per-token FP8: 6.3 bytes
• Per request (average 3,200 tokens): 3,200 × 6.3 = 20.2 KB
• 100 requests total: 100 × 20.2 KB = 2.02 MB

KV cache occupies only 0.0006% of the 356 GB budget. V4 Flash's KV cache is virtually nonexistent on 8×H100. The bottleneck is entirely prefill computation and MoE expert loading bandwidth.

Effective Batch Size: 100 requests can all run in parallel during decode; batch size = 100. In fact, since KV cache barely consumes space, the actual batch size ceiling depends on how many active weight copies can fit in HBM—theoretically supporting thousands of concurrent requests.

8.4 Strategy C: Prefix Tree + Reference Counting

Using Radix Tree to manage shared prefixes.

Prefill Compute Saved by Prefix Reuse:

• Actual prefill tokens needed: 320,000 − 196,000 = 124,000 tokens
• Prefill FLOP: 124,000 × 26 GFLOP = 3.22 PFLOP
• Prefill time: 3.22 / 16 ≈ 0.20s (61% shorter than Strategy A's 0.52s)
• Actual with MoE overhead: approximately 0.3–0.4s

TTFT (Time to First Token) Comparison:

• Strategy A: New request waits for full prefill. 2K context TTFT ≈ 2,000 × 26 GFLOP / (16 PFLOP/s / 100) ≈ 3.25ms (100-way parallel). But actual TTFT ~30–60ms due to intra-batch contention for MoE expert loading bandwidth.
• Strategy C: 80% of system prompt + few-shot can be read directly from cache; actual prefill only computes the user-unique portion (~500 tokens for short requests). TTFT drops to 10–20ms, a 60–70% reduction.

Prefix Cache Storage Cost:

• System Prompt KV cache: 1,500 × 6.3 bytes = 9.5 KB
• 5 groups Few-shot KV cache: 5 × 500 × 6.3 bytes = 15.8 KB
• Total: 25.3 KB
• HBM occupied: negligible

8.5 Economics: $/Mtoken

Estimated using 8×H100 server hourly rental cost (cloud market price approximately $12/hr/GPU; 8 GPUs = $96/hr):

Decode Cost (same for both strategies):

• 13B active parameters; per-token decode FLOP ≈ 2 × 13B = 26 GFLOP
• Batch size 100; each step generates 100 tokens
• Per-step decode time: 26 GFLOP × 100 / 16 PFLOP/s = 0.16ms (theoretical)
• Actual decode speed (including MoE expert loading, CSA selection overhead): approximately 50–70 tokens/s/request (note: the ~200 tok/s from Article 1 is the decode-only theoretical value; 50–70 here is the end-to-end value including prefill contention + MoE all-to-all + actual scheduling; the 3–4× gap comes from production system non-ideal factors)
• 100 concurrent × 60 tokens/s = 6,000 tokens/s aggregate throughput
• Hourly generation: 6,000 × 3,600 = 21.6M tokens
• Decode cost: $96 / 21.6M = $4.4/Mtoken

Prefill Cost (main difference between strategies):

Combined $/Mtoken:

• Strategy A: approximately $6–7/Mtoken (prefill + decode mixed)
• Strategy C: approximately $4.5–5.5/Mtoken
• Difference of approximately 20–25%—Prefix sharing ROI remains significant. This estimate is conservative, not accounting for indirect decode-side benefits from tighter batch scheduling after prefix hits

Benchmarking against V4 Flash official pricing: ¥2/Mtoken (approximately $0.28/Mtoken). Our 8×H100 deployment cost is far higher than the official price—indicating that DeepSeek achieves significantly lower unit costs than an H100 cluster through Ascend 950PR hardware cost advantages (unit price approximately 1/3.6 of H200) + larger cluster scale + more aggressive PD separation. H100 deployment of V4 Flash is not competitive—but the case study reveals the cost structure.

8.6 Case Summary

Core Finding: V4 Flash's KV cache is so small it is no longer a bottleneck in any sense—100 concurrent requests use only 2 MB, not even a rounding error against HBM. The real battlefield has shifted to three places:

1. Prefill compute savings: Prefix sharing is the highest-ROI optimization. Every prefill saved equals 26 GFLOP/token saved. KV cache is free to store, but prefill computation hasn't gotten a cent cheaper.
2. MoE expert loading bandwidth: 284B total parameters distributed across 8 GPUs; each decode step needs to load routed experts. This is the primary source of V4 Flash inference latency.
3. PCIe bandwidth in PD-decoupled architecture: Expert parameter transfer between P nodes and D nodes (not KV cache transfer—the latter is negligible)

This is consistent with the DualPath paper's conclusion—at 98.7% hit rate, the inference bottleneck has shifted from "compute" to "storage I/O bandwidth." On V4 Flash, it has further shifted to "MoE expert loading bandwidth."

IX. Design Decision Matrix

Summarizing the above analysis into a V4-era design decision framework:

X. Open Questions

1. Predictability of CSA Selection Patterns: CSA's top-k selection is determined by model weights and is deterministic for given inputs. If we could offline-analyze CSA's selection patterns (which KV entries are high-frequency selected), it could guide prefix cache eviction strategy. But this requires instrumentation deep into model internals, which current inference frameworks (vLLM, SGLang) don't support.

2. Joint MoE × KV Cache Scheduling: In V4 Flash, each decode step activates 13B parameters (routed experts). The activated experts' parameters and corresponding layers' KV cache are "simultaneously hot"—the two could be jointly scheduled. But on V4, KV cache is so small that joint scheduling's value mainly lies in bandwidth allocation between expert parameter preloading and KV cache reads.

3. Optimal Routing in PD-Decoupled Architecture: After P node completes prefill, it needs to transfer KV cache to D node. V4 Flash's KV cache is tiny (20 KB/request); the transfer itself is not the issue. But choosing which D node—considering D node's MoE expert distribution and current load—is a meaningful scheduling problem.

4. New Bottlenecks at Ultra-Long Context (1M tokens): Although V4 Flash's KV cache is only 6.3 bytes/token, at 1M tokens the total KV cache = 6.3 MB—still not large. The real challenge is CSA's selection precision degradation at 1M length, and HCA's inter-block summary information loss at ultra-long sequences. This is an architecture-level problem, not something scheduling can solve.

XI. KV Cache Delta Encoding: An Orthogonal Optimization Above CSA+HCA

The most worth-elaborating technical direction among the open questions is KV cache delta encoding. If feasible, it could further compress KV cache on top of CSA+HCA.

But first, an honest acknowledgment of a V4-era awkwardness: CSA+HCA has already compressed KV cache to extremely low levels; delta encoding's marginal benefit may be diminishing.

11.1 The Problem

All current KV cache management strategies—whether LRU, Attention Score, or Prefix Tree—are built on the same assumption: each token's KV vector is independent, complete, and incompressible. Quantization schemes (FP8→INT4→TurboQuant's polar compression) compromise on precision but don't change the "store independently per-token" paradigm.

CSA+HCA changed part of this paradigm—CSA does sparse selection at the architecture level, HCA does block-level compression. But the selected retained KV entries are still stored as independent, complete vectors. Delta encoding targets exactly this part.

11.2 Technical Principle: Inter-Frame Prediction Analogy

Video compression has a mature paradigm—Inter-frame Prediction:

• I-frame (keyframe): Stored completely
• P-frame (predictive frame): Only stores motion vectors + residuals relative to the previous frame
• B-frame (bidirectional predictive frame): Interpolates using both preceding and following frames

In natural language sequences, semantically contiguous token segments (e.g., consecutive words in a sentence, consecutive lines in code) typically have highly similar KV vectors. If we treat each token's complete KV as an "I-frame," then:

• KV Delta: Store only ΔKV_t = KV_t − KV_{t-1} (residual)
• KV P-Frame: For semantically contiguous segments, store only one "keyframe" + increments for subsequent tokens
• KV B-Frame: For middle segments, interpolate using surrounding keyframes

11.3 Theoretical Benefit Estimation (Based on V4 Flash)

Under V4 Flash CSA+HCA FP8, per-token KV cache is approximately 6.3 bytes. If delta encoding can compress contiguous segment increments to 1/4 of original size (analogous to H.264 P-frame compression ratio):

Comparison with current mainstream compression schemes on V4 Flash:

• CSA+HCA itself (vs standard MHA): ~50,000× compression
• INT4 quantization (vs FP8): 2× compression (3–5% precision loss)
• TurboQuant (Google, 2026-03): 6× compression (zero precision loss)
• Delta encoding (theoretical, stacked on CSA+HCA): additional 5–13× compression

11.4 Diminishing Marginal Returns Assessment

Here we must make an honest judgment.

In the V3 era (MLA FP8 ~31 KB/token), delta encoding's theoretical benefit was 6–12× compression—from 31 KB to 2.5–5 KB. Absolute savings of 26–28 KB/token; for 100 concurrent requests, 8–9 GB of HBM. Significant.

In the V4 Flash era (CSA+HCA FP8 ~6.3 bytes/token), delta encoding's theoretical benefit is similarly 5–13× compression—from 6.3 bytes to 0.5–1.2 bytes. But absolute savings are only 5–6 bytes/token; for 100 concurrent requests, only 1.6–1.9 MB. Saving 1.6 MB on 640 GB of HBM is essentially zero.

Delta encoding's value on V4 lies not in saving storage space, but potentially in:

1. Reducing KV cache transfer volume in PD-decoupled architecture: While V4 Flash's KV cache is already small (2 MB/100 requests), at very large scale (10,000+ concurrent × long context), PD-decoupled network bandwidth may still become a bottleneck. Delta encoding could reduce transfer volume by another order of magnitude.
2. Reducing HBM bandwidth contention: While KV cache footprint is small, each decode step reads the entire KV. V4 Flash: 3,200 tokens × 6.3 bytes = 20 KB/request/step; 100 concurrent = 2 MB/step. After delta encoding: ~0.4 MB/step—reducing HBM bandwidth usage, freeing bandwidth for MoE expert loading.
3. Ultra-long context scenarios: 1M tokens × 6.3 bytes = 6.3 MB/request. 100 × 1M-context requests = 630 MB—starting to matter. Delta encoding could reduce this to ~120 MB, saving 510 MB HBM. In extreme long-context scenarios, delta encoding still has value.

11.5 Current Research State

As of mid-2026, delta encoding applied to standard Transformer KV cache remains in early exploration, but adjacent fields offer important clues:

Clue 1: SJTU dLLM-Cache (2025) Shanghai Jiao Tong University's EPIC Lab proposed dLLM-Cache for diffusion-based language models (dLLMs), with a highly similar core idea—identify features that change minimally between adjacent denoising steps and only update the parts that change significantly. Measured on LLaDA, Dream, and similar models: up to 9.1× inference speedup with zero precision loss. This proves that "inter-step KV redundancy" is exploitable in non-autoregressive models.

Clue 2: Mamba/RWKV's Implicit Delta State space models (Mamba, RWKV) naturally use recurrent state updates—implicitly storing only "deltas." TTT-Linear (Stanford, 2024) defines the hidden state itself as a learnable model, essentially an extreme form of delta compression. These architectures prove that "not storing complete historical KV, only compressed state" is feasible in principle.

Clue 3: Attention Score Sparsity Research H2O, PyramidKV, and related works show that in long contexts, attention scores are highly concentrated on a small number of tokens—95%+ of KV vectors contribute below the noise threshold to final attention output. This indirectly proves that a large number of KV vectors are "compressible redundancy." CSA leverages this at the architecture level—its top-k selection is essentially H2O's idea固化 solidified into model weights.

Clue 4: FlashMemory-DeepSeek-V4 Paper (2026) DeepSeek published the FlashMemory paper alongside V4, describing the production KV cache management framework. The paper confirms CSA+HCA's KV cache compression ratios and presents end-to-end latency data on Ascend 950PR hardware. This is the public literature closest to V4 production data.

Gap: Applying delta encoding directly to standard Transformer's autoregressive KV cache, on top of CSA+HCA architecture—there is currently no published scheme. This is a clear research gap, but its commercial value at conventional context lengths (<32K) may be limited.

11.6 If Feasible, What It Means (V4 Perspective)

Impact on Inference Economics:

Using V4 Flash as example (8×H100 deployment, 100 concurrent, average 3,200 tokens):

• Current CSA+HCA FP8 KV cache: ~2 MB (negligible)
• After delta encoding (assuming 5× compression): ~0.4 MB (even more negligible)
• The 1.6 MB of HBM saved has no practical significance

But in 1M ultra-long context scenarios:

• Current CSA+HCA FP8 KV cache (100 concurrent × 1M tokens): 630 MB
• After delta encoding: ~126 MB
• The 504 MB saved is meaningful—frees up HBM for MoE experts

Impact on Architecture Evolution:

If delta encoding can achieve 5–13× compression on top of CSA+HCA, then 1M-token ultra-long context KV cache cost drops from ~6 MB/request to ~0.5–1.2 MB/request. This significantly reduces the bandwidth cost (not storage cost) of ultra-long context inference—less KV data to read per decode step, alleviating HBM bandwidth contention.

Impact on Scheduling Strategy:

Delta encoding changes the hot-cold tiering framework. When KV cache is encoded as "I-frame + P-frame sequence":

• I-frames are naturally P0 (must be resident in HBM)
• P-frames are naturally P2/P3 (tiny footprint, negligible transfer cost)
• PD-decoupled KV transfer volume further reduced

But on V4 Flash, this improvement's absolute value is too small to be perceptible in conventional scenarios.

Original Judgment:

Delta encoding was an underappreciated direction in the V3 era; in the V4 era, it faces the awkwardness of diminishing marginal returns.

CSA+HCA has already done aggressive KV compression at the architecture level—sparse selection + block-level compression + conditional memory. Delta encoding compresses further on top of an already tiny base; absolute gains are limited. But delta encoding and CSA+HCA are orthogonal—they act on different dimensions (CSA on "which KV is read"; delta encoding on "how read KV is encoded"). In ultra-long context (>128K) scenarios, the value of orthogonal stacking still exists.

A direction more worth watching is offline analysis of CSA selection patterns—if CSA's top-k selection patterns under different inputs can be modeled, it would be possible to predict which KV entries will be selected without running the model. This has more engineering value than delta encoding, because it directly informs prefix cache eviction strategy and PD-decoupled routing decisions.

Delta encoding is not the silver bullet for the V4 era. The real V4-era battleground is MoE expert loading bandwidth, PD-decoupled routing optimization, and maximizing prefill compute reuse.

References

• DeepSeek V4 Technical Report (2026-04) — V4 Flash (284B/13B) and V4 Pro (1.6T/49B) architecture, CSA+HCA hybrid attention, KV Cache reduced to 7%/10% of V3.2
• FlashMemory-DeepSeek-V4 Paper (2026) — V4 production KV cache management framework, CSA+HCA end-to-end latency data
• Liu et al. "Lost in the Middle: How Language Models Use Long Contexts" (attention sink phenomenon)
• vLLM / PagedAttention Paper (Kwon et al., 2023) — Block-level virtual memory management, memory utilization ~60%→95%+
• SGLang / RadixAttention Paper (Zheng et al., 2023) — Prefix tree reuse, cache hit rate improvement 3–5×
• DeepSeek V3 Technical Report (2024) — MLA design, 671B MoE / 37B activation, FP8 (historical baseline)
• DeepSeek DualPath Paper (2026-02) — 660B production model, KV-Cache hit rate 98.7%, throughput improvement 1.87×/1.96× (V4 production validation)
• TurboQuant (Google, 2026-03) — Polar quantization + JL transform, 6× compression zero loss
• dLLM-Cache (SJTU EPIC Lab, 2025) — Diffusion language model KV cache reuse, 9.1× speedup
• TTT-Linear (Stanford, 2024) — Learnable hidden state, linear-complexity long context
• H2O, PyramidKV — Attention Score sparsity, KV pruning
• FlashAttention (Dao et al.) — Source of KV continuity constraints
• Jie Tang et al. LLM Memory Survey (2026-08) — Architecture-level memory classification framework