This episode examines GraniKV, a KV-cache paging scheme for multi-agent workloads where many requests share a very long prompt prefix. It argues that engines like SGLang treat the shared prefix and each request's private suffix the same way, using 16-token pages, even though the two want opposite things from memory. The discussion first covers the background: PagedAttention, RadixAttention prefix reuse, and the point that reuse saves prefill compute but not the repeated memory reads during decode. It then separates two prior kernel ideas, cascade attention, which reads the prefix once to cut memory traffic, and Hydragen, which batches queries into dense matrix multiplies but needs a contiguous prefix. GraniKV's claim is about storage and integration rather than new kernels. It keeps each prefix as one contiguous slab in a reserved "hot" pool, keeps suffixes at single-token granularity in a "cold" pool, and chooses an attention backend at each decode step. The episode tests where the reported 2.16x throughput gain over stock SGLang at a 16K-token prefix comes from, asking whether the layout or the kernel does the work, with a self-built cascade baseline as the comparison. It is useful for listeners who want to understand why prefix-heavy agent serving is memory-bound and how storage layout interacts with attention kernels.
Sources:
1. GraniKV: Asymmetric Granularity KV-Cache Paging for Multi-Agent Systems with Long Shared Prefix — Jinhyun Jeon, Sungjoo Yoo, 2026
http://arxiv.org/abs/2608.15584v1
2. Efficient Memory Management for Large Language Model Serving with PagedAttention — Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+Large+Language+Model+Serving+with+PagedAttention
3. SGLang: Efficient Execution of Structured Language Model Programs — Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, Ying Sheng, 2024
https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs
4. vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention — Ramya Prabhu, Ajay Nayak, Jayashree Mohan, Ramachandran Ramjee, Ashish Panwar, 2024
https://scholar.google.com/scholar?q=vAttention%3A+Dynamic+Memory+Management+for+Serving+LLMs+without+PagedAttention
5. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Vikranth Srivastava, Shu Liu, Yifan Qiao, Yang Zhou, Xingyu Xiang and colleagues (UC San Diego / UC Berkeley), 2024
https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving
6. Hydragen: High-Throughput LLM Inference with Shared Prefixes — Jordan Juravsky, Bradley Brown, Ryan Ehrlich, Daniel Y. Fu, Christopher Ré, Azalia Mirhoseini, 2024
https://scholar.google.com/scholar?q=Hydragen%3A+High-Throughput+LLM+Inference+with+Shared+Prefixes
7. Cascade Inference: Memory-Bandwidth Efficient Shared Prefix Batch Decoding / FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye, Ruihang Lai, Bo-Ru Lu, Chien-Yu Lin, Size Zheng, Lequn Chen, Tianqi Chen, Luis Ceze (and co-authors), 2024-2025
https://scholar.google.com/scholar?q=Cascade+Inference%3A+Memory-Bandwidth+Efficient+Shared+Prefix+Batch+Decoding+%2F+FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving
8. ChunkAttention: Efficient Self-Attention with Prefix-Aware KV Cache and Two-Phase Partition — Lu Ye, Ze Tao, Yong Huang, Yang Li, 2024
https://scholar.google.com/scholar?q=ChunkAttention%3A+Efficient+Self-Attention+with+Prefix-Aware+KV+Cache+and+Two-Phase+Partition
9. Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs — Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian and colleagues (AWS), 2024
https://scholar.google.com/scholar?q=Bifurcated+Attention%3A+Accelerating+Massively+Parallel+Decoding+with+Shared+Prefixes+in+LLMs
10. Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve — Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav S. Gulavani, Alexey Tumanov, Ramachandran Ramjee, 2024
https://scholar.google.com/scholar?q=Taming+Throughput-Latency+Tradeoff+in+LLM+Inference+with+Sarathi-Serve
11. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024
https://scholar.google.com/scholar?q=DistServe%3A+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving
12. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving — Zihao Ye, Lequn Chen, Ruihang Lai, Wuwei Lin, Yineng Zhang, Stephanie Wang, Tianqi Chen, Baris Kasikci, Vinod Grover, Arvind Krishnamurthy, Luis Ceze, 2025
https://scholar.google.com/scholar?q=FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving
13. FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning / Flash-Decoding for long-context inference — Tri Dao (FlashAttention-2); Tri Dao, Daniel Haziza, Francisco Massa, Grigory Sizov (Flash-Decoding), 2023
https://scholar.google.com/scholar?q=FlashAttention-2%3A+Faster+Attention+with+Better+Parallelism+and+Work+Partitioning+%2F+Flash-Decoding+for+long-context+inference
14. FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (including the cascade / multi-level cascade API) — Ye, Chen, Lai, Shao, Zhao, Zheng, Jin, et al., 2025
https://scholar.google.com/scholar?q=FlashInfer%3A+Efficient+and+Customizable+Attention+Engine+for+LLM+Inference+Serving+%28including+the+cascade+%2F+multi-level+cascade+API%29
15. SGLang: Efficient Execution of Structured Language Model Programs (RadixAttention) — Zheng, Yin, Xie, Sun, Huang, Yu, Cao, Kozyrakis, Stoica, Gonzalez, Barrett, Sheng, 2024
https://scholar.google.com/scholar?q=SGLang%3A+Efficient+Execution+of+Structured+Language+Model+Programs+%28RadixAttention%29
16. Efficient Memory Management for LLM Serving with PagedAttention — Kwon, Li, Zhuang, Sheng, Yu, Gonzalez, Zhang, Stoica, 2023
https://scholar.google.com/scholar?q=Efficient+Memory+Management+for+LLM+Serving+with+PagedAttention
17. HotPrefix: Hotness-aware KV Cache Scheduling for Efficient Prefix Sharing in LLM Inference Systems — Li, Gu, Huan, Wang, Yao, Tian, Chen, 2025
https://scholar.google.com/scholar?q=HotPrefix%3A+Hotness-aware+KV+Cache+Scheduling+for+Efficient+Prefix+Sharing+in+LLM+Inference+Systems
18. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Qin, Li, He, Zhang, Wu, Zheng, Xu, 2025
https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving
19. FlashAttention-2 / FlashDecoding (split-KV) — Dao; Dao, Haziza, Massa, Sizov, 2023-2024
https://scholar.google.com/scholar?q=FlashAttention-2+%2F+FlashDecoding+%28split-KV%29
20. Multi-Level / Prefix-sharing attention in other engines (e.g. DeFT, Prefix-sharing tree attention) — Yao et al. (DeFT) and related tree-attention work, 2024
https://scholar.google.com/scholar?q=Multi-Level+%2F+Prefix-sharing+attention+in+other+engines+%28e.g.+DeFT%2C+Prefix-sharing+tree+attention%29
Interactive Visualization: GraniKV: Asymmetric KV-Cache Paging for Shared-Prefix Agents
