Self-Evolving Curriculum: Bandit-Selected Problems for LLM Reasoning RL

This episode examines Self-Evolving Curriculum, a 2025 paper proposing that a small multi-armed bandit can choose which problem categories an LLM trains on during reinforcement learning from verifiable rewards. It explains why GRPO gets no learning signal from problems the model always solves or never solves, since every advantage in the group is zero. It contrasts the bandit approach with fixed easy-to-hard schedules, random shuffling, and DAPO-style dynamic sampling, which pays for rollouts it then discards. The discussion covers the non-stationary bandit setup: each difficulty level or problem type is an arm, and the signal is reused from advantages GRPO already computes. It also looks at the Countdown, Zebra, ARC-1D and MATH experiments, including the point that the "out-of-distribution" claims are cleaner for the puzzles than for the math benchmarks. Listeners get a clear walkthrough of the mechanism and a skeptical look at whether the evidence supports the paper's claims.

Sources: 1. Self-Evolving Curriculum for LLM Reasoning — Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, Ehsan Kamalloo, 2025 http://arxiv.org/abs/2505.14970 2. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009 https://scholar.google.com/scholar?q=Curriculum+Learning 3. Automated Curriculum Learning for Neural Networks — Alex Graves, Marc G. Bellemare, Jacob Menick, Rémi Munos, Koray Kavukcuoglu, 2017 https://scholar.google.com/scholar?q=Automated+Curriculum+Learning+for+Neural+Networks 4. Teacher-Student Curriculum Learning — Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman, 2017 (arXiv); published in IEEE Transactions on Neural Networks and Learning Systems, 2019 https://scholar.google.com/scholar?q=Teacher-Student+Curriculum+Learning 5. Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey — Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, Peter Stone, 2020 https://scholar.google.com/scholar?q=Curriculum+Learning+for+Reinforcement+Learning+Domains%3A+A+Framework+and+Survey 6. Efficient Online Data Mixing For Language Model Pre-Training — Alon Albalak, Liangming Pan, Colin Raffel, William Yang Wang, 2023 https://scholar.google.com/scholar?q=Efficient+Online+Data+Mixing+For+Language+Model+Pre-Training 7. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Qiying Yu et al., 2025 https://scholar.google.com/scholar?q=DAPO%3A+An+Open-Source+LLM+Reinforcement+Learning+System+at+Scale 8. DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM Post-training — Zhenting Wang, Guofeng Cui, Kun Wan, Wentian Zhao, 2025 https://scholar.google.com/scholar?q=DUMP%3A+Automated+Distribution-Level+Curriculum+Learning+for+RL-based+LLM+Post-training 9. Efficient Reinforcement Finetuning via Adaptive Curriculum Learning (AdaRFT) — Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, Jieyu Zhao, 2025 https://scholar.google.com/scholar?q=Efficient+Reinforcement+Finetuning+via+Adaptive+Curriculum+Learning+%28AdaRFT%29 10. Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning — Sanghwan Bae et al., 2025 https://scholar.google.com/scholar?q=Online+Difficulty+Filtering+for+Reasoning+Oriented+Reinforcement+Learning 11. Prioritized Experience Replay — Tom Schaul, John Quan, Ioannis Antonoglou, David Silver, 2015 https://scholar.google.com/scholar?q=Prioritized+Experience+Replay 12. Not All Samples Are Created Equal: Deep Learning with Importance Sampling — Angelos Katharopoulos, François Fleuret, 2018 https://scholar.google.com/scholar?q=Not+All+Samples+Are+Created+Equal%3A+Deep+Learning+with+Importance+Sampling 13. Evolving Curricula with Regret-Based Environment Design (ACCEL) — Jack Parker-Holder, Minqi Jiang, Michael Dennis, et al., 2022 https://scholar.google.com/scholar?q=Evolving+Curricula+with+Regret-Based+Environment+Design+%28ACCEL%29 14. Prioritized Level Replay — Minqi Jiang, Edward Grefenstette, Tim Rocktäschel, 2021 https://scholar.google.com/scholar?q=Prioritized+Level+Replay 15. Dr. GRPO / Understanding R1-Zero-Like Training: A Critical Perspective — Zichen Liu et al., 2025 https://scholar.google.com/scholar?q=Dr.+GRPO+%2F+Understanding+R1-Zero-Like+Training%3A+A+Critical+Perspective 16. Variance-based / Prompt-difficulty selection for RLVR (e.g., Online Difficulty Filtering, GRESO, Prompt Curriculum Learning) — Various, 2025, 2025 https://scholar.google.com/scholar?q=Variance-based+%2F+Prompt-difficulty+selection+for+RLVR+%28e.g.%2C+Online+Difficulty+Filtering%2C+GRESO%2C+Prompt+Curriculum+Learning%29

Interactive Visualization: Self-Evolving Curriculum: Bandit-Selected Problems for LLM Reasoning RL