Expert-Locality-Aware Decode Routing for MoE Serving

This episode examines ELDR (Expert-Locality-Aware Decode Routing), a routing scheme for serving mixture-of-experts models under prefill-decode disaggregation, from researchers at KAIST, Microsoft Research, and the Shanghai Xingyunzhili Artificial Intelligence Institute. The discussion lays out why decode is memory-bandwidth bound rather than compute bound, and how MoE sparsity — which lowers per-token cost for a single request — becomes a liability at batch scale, since latency now depends on the union of experts a batch touches rather than just token count or load. The key insight is that expert activation patterns are structured rather than random: prompts from similar domains or languages route to overlapping experts, and the model's prefill-time gating decisions already preview which experts a request's decode phase will need. ELDR exploits this by using prefill activations as an early signature to route decode requests toward workers with "warm" overlapping experts, targeting time-per-output-token latency specifically, distinct from ordinary load balancing which only tracks request count or capacity. The conversation grounds this in prior work — DistServe's phase-disaggregation argument, and MoE foundations from Switch Transformers and GShard — before probing how well the paper's offline expert-locality clustering, calibrated on a fixed domain mix, generalizes to shifting real-world traffic.

Sources: 1. Expert-Locality-Aware Decode Routing for MoE Serving https://arxiv.org/pdf/2607.00466 2. Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity — William Fedus, Barret Zoph, Noam Shazeer, 2021 https://scholar.google.com/scholar?q=Switch+Transformers:+Scaling+to+Trillion+Parameter+Models+with+Simple+and+Efficient+Sparsity 3. GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding — Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, Zhifeng Chen, 2020 https://scholar.google.com/scholar?q=GShard:+Scaling+Giant+Models+with+Conditional+Computation+and+Automatic+Sharding 4. DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving — Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, Hao Zhang, 2024 https://scholar.google.com/scholar?q=DistServe:+Disaggregating+Prefill+and+Decoding+for+Goodput-optimized+Large+Language+Model+Serving 5. Mooncake: A KV Cache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, Mingxing Zhang, Yongwei Wu, Weimin Zheng, Xinran Xu (Moonshot AI / Kimi), 2024 https://scholar.google.com/scholar?q=Mooncake:+A+KV+Cache-centric+Disaggregated+Architecture+for+LLM+Serving 6. Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns — Abhimanyu Bambhaniya, Geonhwa Jeong, Jason Park, et al., 2026 https://scholar.google.com/scholar?q=Scaling+Multi-Node+Mixture-of-Experts+Inference+Using+Expert+Activation+Patterns 7. Efficient MoE Serving in the Memory-Bound Regime: Balance Activated Experts, Not Tokens (METRO) — Yanpeng Yu, Haiyue Ma, Krish Agarwal, et al., 2025 https://scholar.google.com/scholar?q=Efficient+MoE+Serving+in+the+Memory-Bound+Regime:+Balance+Activated+Experts,+Not+Tokens+(METRO) 8. Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving — Ruoyu Qin, Zheming Li, Weiran He, et al., 2025 https://scholar.google.com/scholar?q=Mooncake:+A+KVCache-centric+Disaggregated+Architecture+for+LLM+Serving 9. DeepSeek-V3 Technical Report / EPLB: Expert Parallelism Load Balancer — DeepSeek-AI, 2024/2025 https://scholar.google.com/scholar?q=DeepSeek-V3+Technical+Report+/+EPLB:+Expert+Parallelism+Load+Balancer

Interactive Visualization: Expert-Locality-Aware Decode Routing for MoE Serving