Decoupled Semantic Compression: KV Cache Eviction with Semantic Localization and Selection

The key-value (KV) cache dominates the memory footprint of long-context large language model (LLM) inference. Under stringent memory budgets, KV cache eviction must retain only a small subset of tokens while preserving downstream generation quality. Existing eviction methods estimate token importance in a single stage by reading attention from all layers or heuristic layer subsets, even though semantic information is concentrated in only a narrow band of transformer layers. This mixes structural and semantic signals before token selection. We propose Decoupled Semantic Compression (DSC), a two-stage KV cache eviction framework that explicitly separates where semantic importance should be read from what tokens should be retained. DSC first identifies a model-specific semantic layer band through offline, query-free profiling, and then performs band-guided token selection to preserve coherent evidence spans while constructing a shared retention mask compatible with standard attention kernels. Experiments on LongBench and Needle-in-a-Haystack with Qwen3-8B show that DSC consistently outperforms existing KV eviction methods, with the average relative improvement over the strongest baseline increasing from 0.35% at a 30% retention ratio to 24.2% at 1%. DSC also achieves 100% retrieval accuracy on Needle-in-a-Haystack at a 5% retention ratio.

preprint