Emergent Tool Use From Multi-Agent Autocurricula in Hide-and-Seek

This episode examines "Emergent Tool Use From Multi-Agent Autocurricula," in which hiders and seekers in a physics simulator develop strategies like building box forts, using ramps, and locking objects. All of this comes from a reward that only says whether a hider is visible. It sets the work against hand-written rewards and curiosity-driven baselines such as count-based exploration and Random Network Distillation. It also explains the setup: team-based self-play with PPO, movable boxes and ramps, and a head start for the hiders. The discussion asks whether the claimed six phases are measured or only watched. It looks at the statistics the authors track, such as box movement and locking, and notes that the paper mentions Elo and TrueSkill but doesn't run them. It also covers transfer tasks as a way to test what the agents learned. Listeners get a skeptical look at how self-play progress is measured, plus a clear account of why co-adapting agents make that hard.

Sources: 1. Emergent Tool Use From Multi-Agent Autocurricula in Hide-and-Seek https://arxiv.org/pdf/1909.07528 2. Autocurricula and the Emergence of Innovation from Social Interaction: A Manifesto for Multi-Agent Intelligence Research — Joel Z. Leibo, Edward Hughes, Marc Lanctot, Thore Graepel, 2019 https://scholar.google.com/scholar?q=Autocurricula+and+the+Emergence+of+Innovation+from+Social+Interaction%3A+A+Manifesto+for+Multi-Agent+Intelligence+Research 3. Emergent Complexity via Multi-Agent Competition — Trapit Bansal, Jakub Pachocki, Szymon Sidor, Ilya Sutskever, Igor Mordatch, 2018 https://scholar.google.com/scholar?q=Emergent+Complexity+via+Multi-Agent+Competition 4. Grandmaster level in StarCraft II using multi-agent reinforcement learning — Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, et al. (DeepMind), 2019 https://scholar.google.com/scholar?q=Grandmaster+level+in+StarCraft+II+using+multi-agent+reinforcement+learning 5. Dota 2 with Large Scale Deep Reinforcement Learning — OpenAI: Christopher Berner, Greg Brockman, Brooke Chan, et al., 2019 https://scholar.google.com/scholar?q=Dota+2+with+Large+Scale+Deep+Reinforcement+Learning 6. Emergent Coordination Through Competition — Siqi Liu, Guy Lever, Josh Merel, Saran Tunyasuvunakool, Nicolas Heess, Thore Graepel, 2019 https://scholar.google.com/scholar?q=Emergent+Coordination+Through+Competition 7. Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments (MADDPG) — Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, Igor Mordatch, 2017 https://scholar.google.com/scholar?q=Multi-Agent+Actor-Critic+for+Mixed+Cooperative-Competitive+Environments+%28MADDPG%29 8. Emergence of Grounded Compositional Language in Multi-Agent Populations — Igor Mordatch, Pieter Abbeel, 2018 https://scholar.google.com/scholar?q=Emergence+of+Grounded+Compositional+Language+in+Multi-Agent+Populations 9. Evolving 3D Morphology and Behavior by Competition — Karl Sims, 1994 https://scholar.google.com/scholar?q=Evolving+3D+Morphology+and+Behavior+by+Competition 10. Exploration by Random Network Distillation — Yuri Burda, Harrison Edwards, Amos Storkey, Oleg Klimov, 2018 (ICLR 2019) https://scholar.google.com/scholar?q=Exploration+by+Random+Network+Distillation 11. Curiosity-driven Exploration by Self-supervised Prediction — Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, Trevor Darrell, 2017 https://scholar.google.com/scholar?q=Curiosity-driven+Exploration+by+Self-supervised+Prediction 12. Large-Scale Study of Curiosity-Driven Learning — Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, Alexei A. Efros, 2018 https://scholar.google.com/scholar?q=Large-Scale+Study+of+Curiosity-Driven+Learning 13. Unifying Count-Based Exploration and Intrinsic Motivation — Marc G. Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, Rémi Munos, 2016 https://scholar.google.com/scholar?q=Unifying+Count-Based+Exploration+and+Intrinsic+Motivation 14. How transferable are features in deep neural networks? — Jason Yosinski, Jeff Clune, Yoshua Bengio, Hod Lipson, 2014 https://scholar.google.com/scholar?q=How+transferable+are+features+in+deep+neural+networks%3F 15. Quantifying Generalization in Reinforcement Learning — Karl Cobbe, Oleg Klimov, Chris Hesse, Taehoon Kim, John Schulman, 2019 https://scholar.google.com/scholar?q=Quantifying+Generalization+in+Reinforcement+Learning 16. On the Measure of Intelligence — François Chollet, 2019 https://scholar.google.com/scholar?q=On+the+Measure+of+Intelligence 17. A Survey of Zero-shot Generalisation in Deep Reinforcement Learning — Robert Kirk, Amy Zhang, Edward Grefenstette, Tim Rocktäschel, 2021 (JAIR 2023) https://scholar.google.com/scholar?q=A+Survey+of+Zero-shot+Generalisation+in+Deep+Reinforcement+Learning 18. Open-Ended Learning Leads to Generally Capable Agents (XLand) — Open-Ended Learning Team (DeepMind: Stooke, Mahajan, Barros, et al.), 2021 https://scholar.google.com/scholar?q=Open-Ended+Learning+Leads+to+Generally+Capable+Agents+%28XLand%29 19. Grandmaster level in StarCraft II using multi-agent reinforcement learning (AlphaStar league) — Vinyals et al., 2019 https://scholar.google.com/scholar?q=Grandmaster+level+in+StarCraft+II+using+multi-agent+reinforcement+learning+%28AlphaStar+league%29 20. Real World Games Look Like Spinning Tops — Czarnecki, Gidel, Tracey, Tuyls, Omidshafiei, Balduzzi, Jaderberg, 2020 https://scholar.google.com/scholar?q=Real+World+Games+Look+Like+Spinning+Tops 21. Open-ended Learning in Symmetric Zero-sum Games — Balduzzi, Garnelo, Bachrach, Czarnecki, Perolat, Jaderberg, Graepel, 2019 https://scholar.google.com/scholar?q=Open-ended+Learning+in+Symmetric+Zero-sum+Games 22. Paired Open-Ended Trailblazer (POET) and Enhanced POET — Wang, Lehman, Clune, Stanley (2019); Wang, Lehman, Rawal, Zhi, Li, Clune, Stanley (2020), 2019/2020 https://scholar.google.com/scholar?q=Paired+Open-Ended+Trailblazer+%28POET%29+and+Enhanced+POET 23. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design (PAIRED) — Dennis, Jaques, Vinitsky, Bayen, Russell, Critch, Levine, 2020 https://scholar.google.com/scholar?q=Emergent+Complexity+and+Zero-shot+Transfer+via+Unsupervised+Environment+Design+%28PAIRED%29 24. Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play — Sukhbaatar, Lin, Kostrikov, Synnaeve, Szlam, Fergus, 2018 https://scholar.google.com/scholar?q=Intrinsic+Motivation+and+Automatic+Curricula+via+Asymmetric+Self-Play 25. Go-Explore: a New Approach for Hard-Exploration Problems / First return, then explore — Ecoffet, Huizinga, Lehman, Stanley, Clune, 2019/2021 https://scholar.google.com/scholar?q=Go-Explore%3A+a+New+Approach+for+Hard-Exploration+Problems+%2F+First+return%2C+then+explore 26. Never Give Up: Learning Directed Exploration Strategies (NGU) and Agent57 — Badia et al., 2020 https://scholar.google.com/scholar?q=Never+Give+Up%3A+Learning+Directed+Exploration+Strategies+%28NGU%29+and+Agent57 27. Diversity is All You Need: Learning Skills without a Reward Function (DIAYN) — Eysenbach, Gupta, Ibarz, Levine, 2019 https://scholar.google.com/scholar?q=Diversity+is+All+You+Need%3A+Learning+Skills+without+a+Reward+Function+%28DIAYN%29 28. The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (MAPPO) — Yu, Velu, Vinitsky, Gao, Wang, Bayen, Wu, 2021/2022 https://scholar.google.com/scholar?q=The+Surprising+Effectiveness+of+PPO+in+Cooperative+Multi-Agent+Games+%28MAPPO%29 29. Specification gaming: the flip side of AI ingenuity / The Surprising Creativity of Digital Evolution — Krakovna et al. (2020); Lehman et al. (2018), 2018/2020 https://scholar.google.com/scholar?q=Specification+gaming%3A+the+flip+side+of+AI+ingenuity+%2F+The+Surprising+Creativity+of+Digital+Evolution 30. Embodied Intelligence via Learning and Evolution — Gupta, Savarese, Ganguli, Fei-Fei, 2021 https://scholar.google.com/scholar?q=Embodied+Intelligence+via+Learning+and+Evolution 31. SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn RL; Absolute Zero: Reinforced Self-play Reasoning with Zero Data — Liu et al. (SPIRAL); Zhao et al. (Absolute Zero), 2025 https://scholar.google.com/scholar?q=SPIRAL%3A+Self-Play+on+Zero-Sum+Games+Incentivizes+Reasoning+via+Multi-Agent+Multi-Turn+RL%3B+Absolute+Zero%3A+Reinforced+Self-play+Reasoning+with+Zero+Data

Interactive Visualization: Emergent Tool Use From Multi-Agent Autocurricula in Hide-and-Seek