Code Predicts, Jev Judges: Separating Simulation from Evaluation

This episode examines a paper on a closed, API-only action-selection model (Jev) that returns a probability for each option in a single call, with no reasoning text. The paper's question is which decisions such a model can be trusted with. It separates "evaluation" decisions, where the answer can be read from the input and options, from "simulation" decisions, where the right option depends on an unstated prediction such as an opponent's move or a required subgoal like washing a knife before placing it. The discussion works through the paper's test setups, random three-by-three payoff games with iterated dominance depth and the ALFWorld household text game. It also covers the "lure" idea: a fixed surface shortcut that sometimes points to the wrong option, which separates real competence from luck. The paper's proposed fix is to have code do the predicting and the model do the judging. Listeners get a clear framework for why a model can ace trap questions yet fail a simple household step, and for what that means for building cheap, fast agent loops.

Sources: 1. Code Owns the Simulation, Jev Owns the Evaluation — Yaodong Yang, Hongyao Tang, Yi Ma, Xingyu Fan, Weixun Wang, Jinpeng Li, Tianpei Yang, 2026 http://arxiv.org/abs/2610.01834 2. Thinking, Fast and Slow (book) / Dual-Process Theories of Higher Cognition: Advancing the Debate — Daniel Kahneman (2011); Jonathan St. B. T. Evans and Keith E. Stanovich (2013), 2011 / 2013 https://scholar.google.com/scholar?q=Thinking%2C+Fast+and+Slow+%28book%29+%2F+Dual-Process+Theories+of+Higher+Cognition%3A+Advancing+the+Debate 3. Cognitive Reflection and Decision Making (the Cognitive Reflection Test); Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in ChatGPT — Shane Frederick (2005); Thilo Hagendorff, Sarah Fabi, Michal Kosinski (2023), 2005 / 2023 https://scholar.google.com/scholar?q=Cognitive+Reflection+and+Decision+Making+%28the+Cognitive+Reflection+Test%29%3B+Human-like+intuitive+behavior+and+reasoning+biases+emerged+in+large+language+models+but+disappeared+in+ChatGPT 4. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (SayCan) — Michael Ahn, Anthony Brohan, et al. (Google Robotics / Everyday Robots), 2022 https://scholar.google.com/scholar?q=Do+As+I+Can%2C+Not+As+I+Say%3A+Grounding+Language+in+Robotic+Affordances+%28SayCan%29 5. SwiftSage: A Generative Agent with Fast and Slow Thinking for Complex Interactive Tasks — Bill Yuchen Lin, Yicheng Fu, Karina Yang, et al., 2023 https://scholar.google.com/scholar?q=SwiftSage%3A+A+Generative+Agent+with+Fast+and+Slow+Thinking+for+Complex+Interactive+Tasks 6. Measuring and Narrowing the Compositionality Gap in Language Models — Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis, 2022 https://scholar.google.com/scholar?q=Measuring+and+Narrowing+the+Compositionality+Gap+in+Language+Models 7. Can Large Language Models Serve as Rational Players in Game Theory? A Systematic Analysis; Playing Repeated Games with Large Language Models — Caoyun Fan, Jindou Chen, Yaohui Jin, Hao He (2023); Elif Akata, Lion Schulz, et al. (2023), 2023 https://scholar.google.com/scholar?q=Can+Large+Language+Models+Serve+as+Rational+Players+in+Game+Theory%3F+A+Systematic+Analysis%3B+Playing+Repeated+Games+with+Large+Language+Models 8. LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks (and On the Planning Abilities of Large Language Models) — Subbarao Kambhampati et al. (2024); Karthik Valmeekam et al. (2023), 2023-2024 https://scholar.google.com/scholar?q=LLMs+Can%27t+Plan%2C+But+Can+Help+Planning+in+LLM-Modulo+Frameworks+%28and+On+the+Planning+Abilities+of+Large+Language+Models%29 9. Mind's Eye: Grounded Language Model Reasoning through Simulation — Ruibo Liu, Jason Wei, Shixiang Shane Gu, et al., 2022 https://scholar.google.com/scholar?q=Mind%27s+Eye%3A+Grounded+Language+Model+Reasoning+through+Simulation 10. Reasoning with Language Model is Planning with World Model (RAP) — Shibo Hao, Yi Gu, Haodi Ma, et al., 2023 https://scholar.google.com/scholar?q=Reasoning+with+Language+Model+is+Planning+with+World+Model+%28RAP%29 11. Code as Policies: Language Model Programs for Embodied Control; Language to Rewards for Robotic Skill Synthesis — Jacky Liang, Wenlong Huang, et al. (2022); Wenhao Yu, Nimrod Gileadi, et al. (2023), 2022-2023 https://scholar.google.com/scholar?q=Code+as+Policies%3A+Language+Model+Programs+for+Embodied+Control%3B+Language+to+Rewards+for+Robotic+Skill+Synthesis 12. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (MuZero) — Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, David Silver, et al., 2020 https://scholar.google.com/scholar?q=Mastering+Atari%2C+Go%2C+Chess+and+Shogi+by+Planning+with+a+Learned+Model+%28MuZero%29 13. ALFWorld: Aligning Text and Embodied Environments for Interactive Learning — Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, Matthew Hausknecht, 2021 https://scholar.google.com/scholar?q=ALFWorld%3A+Aligning+Text+and+Embodied+Environments+for+Interactive+Learning 14. ReAct: Synergizing Reasoning and Acting in Language Models — Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, Yuan Cao, 2022 https://scholar.google.com/scholar?q=ReAct%3A+Synergizing+Reasoning+and+Acting+in+Language+Models 15. Reflexion: Language Agents with Verbal Reinforcement Learning — Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, Shunyu Yao, 2023 https://scholar.google.com/scholar?q=Reflexion%3A+Language+Agents+with+Verbal+Reinforcement+Learning 16. Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models (LATS) — Andy Zhou, Kai Yan, Michal Shlapentokh-Rothman, Haohan Wang, Yu-Xiong Wang, 2023 https://scholar.google.com/scholar?q=Language+Agent+Tree+Search+Unifies+Reasoning%2C+Acting%2C+and+Planning+in+Language+Models+%28LATS%29 17. Faith and Fate: Limits of Transformers on Compositionality — Dziri et al., 2023 https://scholar.google.com/scholar?q=Faith+and+Fate%3A+Limits+of+Transformers+on+Compositionality 18. Do Large Language Models Latently Perform Multi-Hop Reasoning? — Yang et al., 2024 https://scholar.google.com/scholar?q=Do+Large+Language+Models+Latently+Perform+Multi-Hop+Reasoning%3F 19. The Expressive Power of Transformers with Chain of Thought / Chain of Thought Empowers Transformers to Solve Inherently Serial Problems — Merrill & Sabharwal; Li et al., 2023-2024 https://scholar.google.com/scholar?q=The+Expressive+Power+of+Transformers+with+Chain+of+Thought+%2F+Chain+of+Thought+Empowers+Transformers+to+Solve+Inherently+Serial+Problems 20. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (MuZero) / Mastering the game of Go without human knowledge (AlphaZero) — Schrittwieser et al.; Silver et al., 2020 / 2017 https://scholar.google.com/scholar?q=Mastering+Atari%2C+Go%2C+Chess+and+Shogi+by+Planning+with+a+Learned+Model+%28MuZero%29+%2F+Mastering+the+game+of+Go+without+human+knowledge+%28AlphaZero%29 21. Large Language Models Still Can't Plan / On the Planning Abilities of Large Language Models (PlanBench) — Valmeekam et al., 2023 https://scholar.google.com/scholar?q=Large+Language+Models+Still+Can%27t+Plan+%2F+On+the+Planning+Abilities+of+Large+Language+Models+%28PlanBench%29 22. Language Models Don't Always Say What They Think / Language Models (Mostly) Know What They Know — Turpin et al.; Kadavath et al., 2023 / 2022 https://scholar.google.com/scholar?q=Language+Models+Don%27t+Always+Say+What+They+Think+%2F+Language+Models+%28Mostly%29+Know+What+They+Know 23. Generative Verifiers: Reward Modeling as Next-Token Prediction — Zhang et al., 2024 https://scholar.google.com/scholar?q=Generative+Verifiers%3A+Reward+Modeling+as+Next-Token+Prediction 24. Large Language Models Are Not Robust Multiple Choice Selectors — Zheng et al., 2024 https://scholar.google.com/scholar?q=Large+Language+Models+Are+Not+Robust+Multiple+Choice+Selectors 25. Dreamer / DreamerV3 — Hafner et al., 2020-2023 https://scholar.google.com/scholar?q=Dreamer+%2F+DreamerV3

Interactive Visualization: Code Predicts, Jev Judges: Separating Simulation from Evaluation