Harness Learning: Training Small Models to Rewrite Code Around Frozen Solvers

This episode explores a paper on harness learning, in which a small proposer model is trained with reinforcement learning to rewrite the code wrapped around a frozen solver. The aim is for the system to adapt to unseen tasks with no weight updates. It defines a harness as the program that controls what the model sees, which tools it uses, and how outputs are validated and retried. It contrasts the approach with search-based optimizers such as DSPy, where the optimizer is just a prompted model. The discussion frames the method as loose meta-learning, with code edits standing in for gradient steps in the style of MAML and RL-squared. It also covers GRPO training on verifiable rewards and a Reasoning Gym testbed with held-out puzzle families. Listeners get a clear account of the paper's three questions: whether learned revisions generalize, whether repeated revision helps, and whether training on whole revision sequences beats single edits. The episode also sets up two critiques, including "Harness Updating Is Not Harness Benefit," as lenses for judging the evidence.

Sources: 1. Harness Learning Enables Generalizable Test-Time Adaptation — Alvin Zhang, Xuecheng Liu, Zixuan Wang, Fahim Tajwar, Daman Arora, Ruslan Salakhutdinov, Daniel Khashabi, Yuda Song, Andrea Zanette, 2026 http://arxiv.org/abs/2609.35738 2. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines — Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, Christopher Potts, 2023 (ICLR 2024) https://scholar.google.com/scholar?q=DSPy%3A+Compiling+Declarative+Language+Model+Calls+into+Self-Improving+Pipelines 3. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering — John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press, 2024 https://scholar.google.com/scholar?q=SWE-agent%3A+Agent-Computer+Interfaces+Enable+Automated+Software+Engineering 4. Automated Design of Agentic Systems (ADAS) — Shengran Hu, Cong Lu, Jeff Clune, 2024 https://scholar.google.com/scholar?q=Automated+Design+of+Agentic+Systems+%28ADAS%29 5. Reflexion: Language Agents with Verbal Reinforcement Learning — Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao, 2023 https://scholar.google.com/scholar?q=Reflexion%3A+Language+Agents+with+Verbal+Reinforcement+Learning 6. Reasoning Gym: Reasoning Environments for Reinforcement Learning with Verifiable Rewards — Zafir Stojanovski, Oliver Stanley, Joe Sharratt, Richard Jones, Abdulhakeem Adefioye, Jean Kaddour, Andreas Köpf, 2025 https://scholar.google.com/scholar?q=Reasoning+Gym%3A+Reasoning+Environments+for+Reinforcement+Learning+with+Verifiable+Rewards 7. Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks — Chelsea Finn, Pieter Abbeel, Sergey Levine, 2017 https://scholar.google.com/scholar?q=Model-Agnostic+Meta-Learning+for+Fast+Adaptation+of+Deep+Networks 8. RL²: Fast Reinforcement Learning via Slow Reinforcement Learning — Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever, Pieter Abbeel, 2016 https://scholar.google.com/scholar?q=RL%C2%B2%3A+Fast+Reinforcement+Learning+via+Slow+Reinforcement+Learning 9. Learning to Learn by Gradient Descent by Gradient Descent — Marcin Andrychowicz, Misha Denil, Sergio Gómez Colmenarejo, Matthew W. Hoffman, David Pfau, Tom Schaul, Matthew Botvinick, Nando de Freitas, 2016 https://scholar.google.com/scholar?q=Learning+to+Learn+by+Gradient+Descent+by+Gradient+Descent 10. Mathematical Discoveries from Program Search with Large Language Models (FunSearch) — Bernardino Romera-Paredes, Mohammadamin Barekatain, Alexander Novikov, Matej Balog, M. Pawan Kumar, Emilien Dupont, Francisco J. R. Ruiz, Jordan S. Ellenberg, Pengming Wang, Omar Fawzi, Pushmeet Kohli, Alhussein Fawzi, 2023 (Nature 2023/2024) https://scholar.google.com/scholar?q=Mathematical+Discoveries+from+Program+Search+with+Large+Language+Models+%28FunSearch%29 11. Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories — Shao et al., 2026 https://scholar.google.com/scholar?q=Harness-R1%3A+Learning+to+Edit+Executable+Runtime+Harnesses+from+Agent+Failure+Trajectories 12. Rethinking the Evaluation of Harness Evolution for Agents — Wang et al., 2026 https://scholar.google.com/scholar?q=Rethinking+the+Evaluation+of+Harness+Evolution+for+Agents 13. Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents — Lin et al., 2026 https://scholar.google.com/scholar?q=Harness+Updating+Is+Not+Harness+Benefit%3A+Disentangling+Evolution+Capabilities+in+Self-Evolving+LLM+Agents 14. Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0 — Wang, Kattakinda, Feizi, 2026 https://scholar.google.com/scholar?q=Do+Agent+Optimizers+Compound%3F+A+Continual-Learning+Evaluation+on+Terminal-Bench+2.0 15. Evo-Bench: Can Language Models Improve Agent Harness? — Huang et al., 2026 https://scholar.google.com/scholar?q=Evo-Bench%3A+Can+Language+Models+Improve+Agent+Harness%3F 16. Meta-Harness: End-to-End Optimization of Model Harnesses — Lee, Nair, Zhang, Lee, Khattab, Finn, 2026 https://scholar.google.com/scholar?q=Meta-Harness%3A+End-to-End+Optimization+of+Model+Harnesses 17. AutoHarness: improving LLM agents by automatically synthesizing a code harness — Lou, Lázaro-Gredilla, Dedieu, Wendelken, Lehrach, Murphy, 2026 https://scholar.google.com/scholar?q=AutoHarness%3A+improving+LLM+agents+by+automatically+synthesizing+a+code+harness 18. WHALE: A simple recipe for joint harness-weight optimization — Kim, Lee, Lee, Finn, Lee, 2026 https://scholar.google.com/scholar?q=WHALE%3A+A+simple+recipe+for+joint+harness-weight+optimization 19. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents — Zhang, Hu, Lu, Lange, Clune, 2025 https://scholar.google.com/scholar?q=Darwin+G%C3%B6del+Machine%3A+Open-Ended+Evolution+of+Self-Improving+Agents 20. Self-Taught Optimizer (STOP): Recursively Self-Improving Code Generation — Zelikman, Lorch, Mackey, Kalai, 2024 https://scholar.google.com/scholar?q=Self-Taught+Optimizer+%28STOP%29%3A+Recursively+Self-Improving+Code+Generation 21. Feedback Friction: LLMs Struggle to Fully Incorporate External Feedback — Jiang, Zhang, Wang, Andrews, Khashabi, 2025 https://scholar.google.com/scholar?q=Feedback+Friction%3A+LLMs+Struggle+to+Fully+Incorporate+External+Feedback 22. AdaptFlow: Adaptive Workflow Optimization via Meta-Learning — Zhu et al., 2025 https://scholar.google.com/scholar?q=AdaptFlow%3A+Adaptive+Workflow+Optimization+via+Meta-Learning 23. Training a Generally Curious Agent — Tajwar, Jiang, Thankaraj, Rahman, Kolter, Schneider, Salakhutdinov, 2025 https://scholar.google.com/scholar?q=Training+a+Generally+Curious+Agent 24. TTHE: Test-time harness evolution — Nie et al., 2026 https://scholar.google.com/scholar?q=TTHE%3A+Test-time+harness+evolution

Interactive Visualization: Harness Learning: Training Small Models to Rewrite Code Around Frozen Solvers