This episode explores Hypic, a system for caching independent prompt segments on hybrid-attention LLMs that don't maintain a conventional per-token KV cache. The discussion breaks down why prefill — the one-time pass over massive RAG and agent prompts — dominates serving cost, and how existing position-independent caching techniques (built on splicing per-token key-value vectors) simply don't apply to linear-attention layers that compress history into a single fixed-size state. It covers how production models like Qwen3.5, MiniMax-M1, Ring-2.5, and Kimi-Linear increasingly rely on this compressed-state attention, and why the paper's authors instead identify a transition operator that lets independently-cached segments combine as though computed sequentially — a genuinely new algebraic approach rather than a workaround. Listeners get a debate over whether hybrid-attention caching is solving a real production gap or a still-theoretical one, along with a preview of the operator mechanics that make segment composition work, illustrated with a failure case from RetNet.
Sources:
1. HYPIC: Accelerating Hybrid-Attention LLM Serving with Position-Independent Caching — Yifei Liu, Juntong Wu, Yang Liu, Junhao Hu, Minghao Li, Xiaoxu Chen, Weihang Chen, 2026
http://arxiv.org/abs/2607.01299
2. Prompt Cache: Modular Attention Reuse for Low-Latency Inference — In Gim, Guojun Chen, et al. (Yale University), 2024
https://scholar.google.com/scholar?q=Prompt+Cache:+Modular+Attention+Reuse+for+Low-Latency+Inference
3. CacheBlend: Fast Large Language Model Serving for RAG with Cached Knowledge Fusion — Jiayi Yao, Hanchen Li, et al. (University of Chicago), 2025 (EuroSys)
https://scholar.google.com/scholar?q=CacheBlend:+Fast+Large+Language+Model+Serving+for+RAG+with+Cached+Knowledge+Fusion
4. Marconi: Prefix Caching for the Era of Hybrid LLMs — Researchers including collaborators from Princeton and Mamba co-author Tri Dao (author list not fully certain — verify before quoting on-air), 2025 (MLSys)
https://scholar.google.com/scholar?q=Marconi:+Prefix+Caching+for+the+Era+of+Hybrid+LLMs
5. RAGCache: Efficient Knowledge Caching for Retrieval-Augmented Generation — Chao Jin, Zili Zhang, et al. (Peking University / Shanghai AI Laboratory), 2024
https://scholar.google.com/scholar?q=RAGCache:+Efficient+Knowledge+Caching+for+Retrieval-Augmented+Generation
6. EPIC: Efficient Position-Independent Caching for Serving Large Language Models — Junhao Hu, Wenrui Huang, Weidong Wang, et al., 2025
https://scholar.google.com/scholar?q=EPIC:+Efficient+Position-Independent+Caching+for+Serving+Large+Language+Models
7. Gated Delta Networks: Improving Mamba2 with Delta Rule — Songlin Yang, Jan Kautz, Ali Hatamizadeh, 2025
https://scholar.google.com/scholar?q=Gated+Delta+Networks:+Improving+Mamba2+with+Delta+Rule
8. Mooncake: Trading More Storage for Less Computation — A KVCache-Centric Architecture for Serving LLM Chatbot — Ruoyu Qin, Zheming Li, Weiran He, et al., 2025
https://scholar.google.com/scholar?q=Mooncake:+Trading+More+Storage+for+Less+Computation+—+A+KVCache-Centric+Architecture+for+Serving+LLM+Chatbot
9. You Need an Encoder for Native Position-Independent Caching — Shiju Zhao, Junhao Hu, Jiaqi Zheng, Guihai Chen, 2026
https://scholar.google.com/scholar?q=You+Need+an+Encoder+for+Native+Position-Independent+Caching
Interactive Visualization: Hypic: Position-Independent KV Caching for Hybrid-Attention LLM Serving
