“Character Training for Risk-Averse Agents” by arav-dhoot, Punya Syon Pandey, Jamie Johnson, Daniel Tan, Elliott Thornley, David Africa

Tl;Dr:

• We might be able to reduce harm from misaligned AI if it was sufficiently risk-averse, as it could be paid to cooperate rather than rebel (Thornley & MacAskill 2026).
• We think character training might be a good way to do this.
• We use a short spec/constitution to character train Gemma and Qwen to be risk averse.
• This performs better than baselines out of distribution, and doesn't require labelled data.
• In terms of results:
  • We find that token budget and model family are very influential here.  
  • We also find that writing the constitution declaratively ("I am...") works better than procedurally ("Whenever X, I...").
  • We didn't find that increasing the number of examples worked well.
  • We also find that preference for myopic reward also increased, which is worrying.
• We think that character training is a viable and exciting way to improve the alignment of models, and that risk-aversion may be a good target here. But, it's important to understand character training well, especially since undesirable behaviours can also leak.

Introduction

Thornley and MacAskill (2026) argue that paying misaligned AIs to cooperate is unaffordable if they are risk-neutral. A sufficiently risk-averse agent, by contrast, has steeply diminishing marginal [...]

---

Outline:

(01:36) Introduction

(02:56) Approach

(04:32) Results

(07:49) Discussion

---

First published:
October 16th, 2026

Source:
https://www.lesswrong.com/posts/HXkQaWpuaxG6fMkTm/character-training-for-risk-averse-agents

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.