Tl;dr: The frontier AI companies’ implicit alignment plans center on rushing towards RSI with something like current models. This plan is (at best) contentious, and many believe it will have catastrophic consequences. Nonetheless, it is the track we are on. It is therefore important to collect evidence which helps us anticipate how this will go.
—
Note: This post is based on conversations inside the Arcadia Impact alignment team, thinking about what constitutes “good” work in third-party alignment research.
What are the labs’ current alignment strategies?
In practice, the labs’ playbook for producing aligned models looks like the following:
Shape the model's character. Both Anthropic and OpenAI treat alignment training as a question of which character or persona the model takes on. Ideally, planting a ‘safe’ character into the model will generalize to all of its other actions.
Apply extensive RL pressure. Somewhat independently of the above, the labs do extraordinary amounts of RL on the model. This increases its capabilities, but also encourages reward-seeking behaviors and metagaming, and may ultimately lead to scheming and other forms of instrumental convergence.
They also might apply various techniques here such as inoculation prompting, confessions, and the like.
Race towards prosaic RSI. The labs intend to use this model for [...]
---
Outline:
(00:46) What are the labs' current alignment strategies?
(02:13) This plan seems terrible
(05:15) Consensus-building is hard
(06:16) We need better evidence
(08:08) How should alignment research change?
(10:10) Specific directions we're excited about
The original text contained 5 footnotes which were omitted from this narration.
Podden och tillhörande omslagsbild på den här sidan tillhör
LessWrong. Innehållet i podden är skapat av LessWrong och inte av,
eller tillsammans med, Poddtoppen.
