This episode examines SAC, a disaggregated KV-cache system built to serve sparse-attention LLMs like DeepSeek-V3.2 by pairing prefill and decode instances with a CXL-based memory pool instead of RDMA. The discussion traces how sparse attention mechanisms — specifically DeepSeek Sparse Attention's Lightning Indexer, which selects only the top-2048 relevant KV entries per layer — break the assumptions behind existing RDMA-based disaggregation systems like Mooncake and LMCache, which haul the entire KV cache across the network regardless of how much actually gets used. It explains why RDMA's message-based protocol can't cheaply serve scattered top-k lookups, while CXL's cache-line-granular load/store semantics can, and walks through why this matters for two concrete failure modes: wasted network bandwidth and wasted local memory. The hosts cite SAC's reported gains over an RDMA baseline — 2.1x throughput, 9.7x lower time-to-first-token, and 1.8x lower time-between-tokens — before scrutinizing the paper's benchmark methodology, questioning whether the RDMA-vs-CXL comparison (loopback NICs versus a single-hop CXL switch) is genuinely apples-to-apples. Listeners interested in the shifting compute-to-memory bottleneck in long-context LLM serving, and how new interconnects are reshaping systems design around sparse attention, will find the critical dissection of the experimental setup especially useful.
Sources:
1. SAC: Disaggregated KV Cache System for Sparse Attention LLMs with CXL — Ruiyang Ma, Teng Ma, Junru Li, Hantian Zha, Xuchun Shang, Qingda Hu, Zheng Liu, Xinjun Yang, Tao Ma, Guojie Luo, 2026
http://arxiv.org/abs/2606.19746
2. Mooncake: Trading More Storage for Less Computation — A KVCache-centric Architecture for Serving LLM Chatbot — Ruoyu Qin, Zheming Li, Weiran He, et al., 2025
https://scholar.google.com/scholar?q=Mooncake%3A+Trading+More+Storage+for+Less+Computation+%E2%80%94+A+KVCache-centric+Architecture+for+Serving+LLM+Chatbot
3. Beluga: A CXL-based Memory Architecture for Scalable and Efficient LLM KVCache Management — Xinjun Yang, Qingda Hu, Junru Li, Feifei Li, et al., 2026
https://scholar.google.com/scholar?q=Beluga%3A+A+CXL-based+Memory+Architecture+for+Scalable+and+Efficient+LLM+KVCache+Management
4. TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale — Dongha Yoon, Younghoon Min, Hoshik Kim, Sam H. Noh, Jongryool Kim, 2025
https://scholar.google.com/scholar?q=TraCT%3A+Disaggregated+LLM+Serving+with+CXL+Shared+Memory+KV+Cache+at+Rack-Scale
5. HiSparse: High-Efficiency Sparse Attention Inference in SGLang — Zhiqiang Xie, Zhangheng Huang, Tingwei Huang, 2026
https://scholar.google.com/scholar?q=HiSparse%3A+High-Efficiency+Sparse+Attention+Inference+in+SGLang
6. SnapKV: LLM Knows What You Are Looking For Before Generation — Yuhong Li, Yingbing Huang, Bowen Yang, et al., 2024
https://scholar.google.com/scholar?q=SnapKV%3A+LLM+Knows+What+You+Are+Looking+For+Before+Generation
7. H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models — Zhenyu Zhang, Ying Sheng, Tianyi Zhou, et al., 2023
https://scholar.google.com/scholar?q=H2O%3A+Heavy-Hitter+Oracle+for+Efficient+Generative+Inference+of+Large+Language+Models
Interactive Visualization: SAC: Making Sparse Attention KV Caches Actually Sparse with CXL
