Compute or Load? Cake's Smart KV Cache Scheduler

This episode examines "Compute Or Load KV Cache? Why not Both?" (Jin, Liu, et al., University of Michigan), which introduces Cake, a scheduling system for LLM inference. The discussion traces the KV cache problem from its roots in the attention mechanism through the rise of prefix caching in production systems like OpenAI, Anthropic, and DeepSeek, then explains why loading a cached prefix isn't automatically cheap — most cache hits land on slow disk tiers rather than fast GPU memory. The core insight covered is that compute cost per chunk rises across a sequence while I/O cost stays flat, which lets Cake run a "meet in the middle" two-pointer scheduler that computes early chunks on GPU while simultaneously loading later chunks from storage. Listeners interested in LLM serving infrastructure will find concrete numbers throughout — like the 30-second prefill tax on a 72,000-token input — that ground the tradeoff between recomputation and I/O in real system constraints rather than abstract theory.

Sources: 1. Compute Or Load KV Cache? Why Not Both? — Shuowei Jin, Xueshen Liu, Qingzhao Zhang, Z. Morley Mao, 2024 http://arxiv.org/abs/2410.03065 2. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Qin, R. et al. (Moonshot AI / Kimi), 2024 https://scholar.google.com/scholar?q=Mooncake%3A+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 3. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model — DeepSeek-AI, 2024 https://scholar.google.com/scholar?q=DeepSeek-V2%3A+A+Strong%2C+Economical%2C+and+Efficient+Mixture-of-Experts+Language+Model 4. CacheBlend: Fast Large Language Model Serving with Cached Knowledge Fusion — Yao, J., Li, H., Liu, Y., et al., 2024 https://scholar.google.com/scholar?q=CacheBlend%3A+Fast+Large+Language+Model+Serving+with+Cached+Knowledge+Fusion 5. Preble: Efficient Distributed Prompt Scheduling for LLM Serving — Srivatsa, V. et al., 2024 https://scholar.google.com/scholar?q=Preble%3A+Efficient+Distributed+Prompt+Scheduling+for+LLM+Serving

Interactive Visualization: Compute or Load? Cake's Smart KV Cache Scheduler