This episode examines the 2022 survey "Emergent Abilities of Large Language Models," which argues that some few-shot abilities sit at chance level in small models and then jump sharply in larger ones. It sets up the contrast with smooth scaling laws for cross-entropy loss. It also traces where "emergence" comes from, namely Anderson's "More Is Different" and the loose phase-transition metaphor, and it shows that compute is only a proxy for scale, since data, architecture and training objective all shift where an ability appears. The discussion goes through the evidence: curves stitched together from separately trained models such as GPT-3, LaMDA, Gopher and PaLM, with only a few points per family. Examples include BIG-Bench arithmetic, TruthfulQA, MMLU, and chain-of-thought prompting on GSM8K, which helps only past roughly 10^23 FLOPs. Listeners get a skeptical look at whether the paper actually demonstrates unpredictability, or only builds it into the definition of emergence.
Sources:
1. Emergent Abilities of Large Language Models — Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus, 2022
http://arxiv.org/abs/2206.07682
2. Emergent Abilities of Large Language Models — Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, William Fedus, 2022
https://scholar.google.com/scholar?q=Emergent+Abilities+of+Large+Language+Models
3. Are Emergent Abilities of Large Language Models a Mirage? — Rylan Schaeffer, Brando Miranda, Sanmi Koyejo, 2023
https://scholar.google.com/scholar?q=Are+Emergent+Abilities+of+Large+Language+Models+a+Mirage%3F
4. Predictability and Surprise in Large Generative Models — Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Scott Johnston, Andy Jones, Nicholas Joseph, Jackson Kernian, Shauna Kravec, Ben Mann, Neel Nanda, Kamal Ndousse, Catherine Olsson, Daniela Amodei, Tom Brown, Jared Kaplan, Sam McCandlish, Christopher Olah, Dario Amodei, Jack Clark, 2022
https://scholar.google.com/scholar?q=Predictability+and+Surprise+in+Large+Generative+Models
5. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models (BIG-bench) — Aarohi Srivastava and 400+ collaborators (including Guy Gur-Ari, Jascha Sohl-Dickstein and Ethan Dyer as core organizers), 2022
https://scholar.google.com/scholar?q=Beyond+the+Imitation+Game%3A+Quantifying+and+Extrapolating+the+Capabilities+of+Language+Models+%28BIG-bench%29
6. More Is Different — Philip W. Anderson, 1972
https://scholar.google.com/scholar?q=More+Is+Different
7. Scaling Laws for Neural Language Models — Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, Dario Amodei, 2020
https://scholar.google.com/scholar?q=Scaling+Laws+for+Neural+Language+Models
8. Training Compute-Optimal Large Language Models (Chinchilla) — Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre, 2022
https://scholar.google.com/scholar?q=Training+Compute-Optimal+Large+Language+Models+%28Chinchilla%29
9. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets — Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin, Vedant Misra, 2022
https://scholar.google.com/scholar?q=Grokking%3A+Generalization+Beyond+Overfitting+on+Small+Algorithmic+Datasets
10. In-context Learning and Induction Heads — Catherine Olsson et al., 2022
https://scholar.google.com/scholar?q=In-context+Learning+and+Induction+Heads
11. Progress Measures for Grokking via Mechanistic Interpretability — Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, Jacob Steinhardt, 2023
https://scholar.google.com/scholar?q=Progress+Measures+for+Grokking+via+Mechanistic+Interpretability
12. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Jason Wei et al., 2022
https://scholar.google.com/scholar?q=Chain-of-Thought+Prompting+Elicits+Reasoning+in+Large+Language+Models
13. Finetuned Language Models Are Zero-Shot Learners (FLAN) — Jason Wei et al., 2022
https://scholar.google.com/scholar?q=Finetuned+Language+Models+Are+Zero-Shot+Learners+%28FLAN%29
14. Show Your Work: Scratchpads for Intermediate Computation with Language Models — Maxwell Nye et al., 2021
https://scholar.google.com/scholar?q=Show+Your+Work%3A+Scratchpads+for+Intermediate+Computation+with+Language+Models
15. Language Models (Mostly) Know What They Know — Saurav Kadavath et al., 2022
https://scholar.google.com/scholar?q=Language+Models+%28Mostly%29+Know+What+They+Know
16. Inverse Scaling Prize / Inverse Scaling: When Bigger Isn't Better — Ian McKenzie et al., 2022–2023
https://scholar.google.com/scholar?q=Inverse+Scaling+Prize+%2F+Inverse+Scaling%3A+When+Bigger+Isn%27t+Better
17. Broken Neural Scaling Laws — Ethan Caballero, Kshitij Gupta, Irina Rish, David Krueger, 2022
https://scholar.google.com/scholar?q=Broken+Neural+Scaling+Laws
18. Transcending Scaling Laws with 0.1% Extra Compute (UL2R) — Yi Tay et al., 2022
https://scholar.google.com/scholar?q=Transcending+Scaling+Laws+with+0.1%25+Extra+Compute+%28UL2R%29
19. Understanding Emergent Abilities of Language Models from the Loss Perspective — Zhengxiao Du et al., 2024
https://scholar.google.com/scholar?q=Understanding+Emergent+Abilities+of+Language+Models+from+the+Loss+Perspective
20. Data Distributional Properties Drive Emergent In-Context Learning in Transformers — Stephanie Chan et al., 2022
https://scholar.google.com/scholar?q=Data+Distributional+Properties+Drive+Emergent+In-Context+Learning+in+Transformers
Interactive Visualization: Emergent Abilities of Large Language Models: Real or Metaphor?
