"Gradient Routing Localizes and Removes Unintended Behaviors in RL" by shawnghu, Jake Ward, ariaw, Nathan Hu, Bryce Woodworth, TurnTrout, cloud - LessWrong (Curated & Popular) | Lyssna här

Authors: Shawn Hu*, Jake Ward*, Aria Wong, Nathan Hu, Bryce Woodworth, Alexander Matt Turner, Alex Cloud (*Equal contribution, randomly ordered)

tl;dr. During training, models sometimes learn to cheat at tasks rather than solve them in the intended way, a problem known as reward hacking. One mitigation is to use an automated monitor to detect and penalize hacks, but this can teach the model to hack in ways the monitor can’t detect. We propose GRAFT, which instead confines the learning updates from detected hacks to a designated set of parameters that we remove after training. We find that removing these parameters can prevent reward hacks, including hacks that the monitors can’t detect. In our experiments, which are limited to small models and simple monitors, this method prevents reward hacks while still teaching the model to perform the intended task.

📄Paper, 💻Code, 🐦Twitter

Research done as part of MATS. This article is cross-posted to the Anthropic Alignment Science Blog.

Background

Reinforcement learning (RL) is an important component of frontier language model training. During RL, a model attempts to complete tasks given by a developer, and attempts are scored by a reward function. Sometimes models can find ways to “cheat” at tasks [...]

---

Outline:

(01:22) Background

[... 4 more sections]

---

First published:
October 19th, 2026

Source:
https://www.lesswrong.com/posts/mKJEBiuuxHKuCoTB6/gradient-routing-localizes-and-removes-unintended-behaviors

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Figure 1. Gradient Routing Adapter Fine-Tuning (GRAFT). Additional neurons are added to each MLP in a decoder-only Transformer; these are partitioned into RETAIN and FORGET. When training in the target environment (left, center), both adapters are enabled for both the rollout phase and the actor update's forward pass. On trajectories not flagged by the monitor (left), both the RETAIN and FORGET adapters receive gradient updates. On trajectories flagged by the monitor (center), only the FORGET adapter is updated and the gradient update to this adapter is amplified by a constant factor. We concurrently train the model in a separate anchor environment where only the RETAIN adapter is enabled and updated (right).

Figure 2. GRAFT localizes and removes reward hacking in Qwen3-8B trained in an arithmetic reasoning environment. Left: GRAFT outperforms all tested methods on reward hack rate and achieves competitive performance on the target task.

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Rss Apple Podcaster