This episode examines "Automatic Curriculum Learning through Value Disagreement," a NeurIPS 2020 paper proposing that a reinforcement learning agent can choose its own training goals. It walks through the background: goal-conditioned RL with sparse rewards, Hindsight Experience Replay as an implicit curriculum, and earlier explicit approaches like Goal GAN and learning-progress teachers, which found that plain uniform sampling is a surprisingly strong baseline. The core idea is to train an ensemble of Q-functions and sample goals in proportion to the standard deviation of their predictions. The reasoning is that the ensemble agrees on easy goals and on hopeless ones, but splits on goals at the frontier of competence. The discussion explains the method as a simple loop: draw candidate goals, score each by ensemble disagreement, and sample one. It needs no adversary or success classifier, and the policy learner never sees the ensemble. Throughout, the hosts treat the disagreement-marks-the-frontier hypothesis as a claim to test rather than accept, which makes this a useful look at what a cheap uncertainty signal can and can't tell a learning agent.
Sources:
1. Automatic Curriculum Learning Through Value Disagreement in Reinforcement Learning
https://proceedings.neurips.cc/paper/2020/file/566f0ea4f6c2e947f36795c8f58ba901-Paper.pdf
2. Curriculum Learning — Yoshua Bengio, Jérôme Louradour, Ronan Collobert, Jason Weston, 2009
https://scholar.google.com/scholar?q=Curriculum+Learning
3. Automatic Goal Generation for Reinforcement Learning Agents (Goal GAN) — Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel, 2018
https://scholar.google.com/scholar?q=Automatic+Goal+Generation+for+Reinforcement+Learning+Agents+%28Goal+GAN%29
4. Teacher-Student Curriculum Learning — Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman, 2017 (IEEE TNNLS 2019)
https://scholar.google.com/scholar?q=Teacher-Student+Curriculum+Learning
5. Automatic Curriculum Learning For Deep RL: A Short Survey — Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, Pierre-Yves Oudeyer, 2020
https://scholar.google.com/scholar?q=Automatic+Curriculum+Learning+For+Deep+RL%3A+A+Short+Survey
6. Universal Value Function Approximators — Tom Schaul, Daniel Horgan, Karol Gregor, David Silver, 2015
https://scholar.google.com/scholar?q=Universal+Value+Function+Approximators
7. Hindsight Experience Replay — Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, Pieter Abbeel, Wojciech Zaremba, 2017 (NeurIPS)
https://scholar.google.com/scholar?q=Hindsight+Experience+Replay
8. Reinforcement Learning with Deep Energy-Based Policies / Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Research — Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, Vikash Kumar, Wojciech Zaremba, 2018
https://scholar.google.com/scholar?q=Reinforcement+Learning+with+Deep+Energy-Based+Policies+%2F+Multi-Goal+Reinforcement+Learning%3A+Challenging+Robotics+Environments+and+Request+for+Research
9. Goal-Conditioned Reinforcement Learning: Problems and Solutions — Minghuan Liu, Menghui Zhu, Weinan Zhang, 2022 (IJCAI survey)
https://scholar.google.com/scholar?q=Goal-Conditioned+Reinforcement+Learning%3A+Problems+and+Solutions
10. Teacher algorithms for curriculum learning of deep RL in continuously parameterized environments — Rémy Portelas, Cédric Colas, Katja Hofmann, Pierre-Yves Oudeyer, 2019
https://scholar.google.com/scholar?q=Teacher+algorithms+for+curriculum+learning+of+deep+RL+in+continuously+parameterized+environments
11. Automatic Goal Generation for Reinforcement Learning Agents — Carlos Florensa, David Held, Xinyang Geng, Pieter Abbeel, 2018
https://scholar.google.com/scholar?q=Automatic+Goal+Generation+for+Reinforcement+Learning+Agents
12. Self-Supervised Exploration via Disagreement — Deepak Pathak, Dhiraj Gandhi, Abhinav Gupta, 2019
https://scholar.google.com/scholar?q=Self-Supervised+Exploration+via+Disagreement
13. Deep Exploration via Bootstrapped DQN — Ian Osband, Charles Blundell, Alexander Pritzel, Benjamin Van Roy, 2016
https://scholar.google.com/scholar?q=Deep+Exploration+via+Bootstrapped+DQN
14. Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles — Balaji Lakshminarayanan, Alexander Pritzel, Charles Blundell, 2017
https://scholar.google.com/scholar?q=Simple+and+Scalable+Predictive+Uncertainty+Estimation+using+Deep+Ensembles
15. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play — Sainbayar Sukhbaatar, Zeming Lin, Ilya Kostrikov, Gabriel Synnaeve, Arthur Szlam, Rob Fergus, 2018
https://scholar.google.com/scholar?q=Intrinsic+Motivation+and+Automatic+Curricula+via+Asymmetric+Self-Play
16. Prioritized Level Replay — Minqi Jiang, Edward Grefenstette, Tim Rocktäschel, 2021
https://scholar.google.com/scholar?q=Prioritized+Level+Replay
17. Emergent Tool Use From Multi-Agent Autocurricula — Bowen Baker et al., 2019
https://scholar.google.com/scholar?q=Emergent+Tool+Use+From+Multi-Agent+Autocurricula
18. Paired Open-Ended Trailblazer (POET) and Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (PAIRED) — Rui Wang, Joel Lehman, Jeff Clune, Kenneth Stanley; Michael Dennis et al., 2019/2020
https://scholar.google.com/scholar?q=Paired+Open-Ended+Trailblazer+%28POET%29+and+Emergent+Complexity+and+Zero-shot+Transfer+via+Unsupervised+Environment+Design+%28PAIRED%29
19. DAPO: An Open-Source LLM Reinforcement Learning System at Scale — Qiying Yu et al., 2025
https://scholar.google.com/scholar?q=DAPO%3A+An+Open-Source+LLM+Reinforcement+Learning+System+at+Scale
Interactive Visualization: Automatic Curriculum Learning Through Value Disagreement in Reinforcement Learning
