“The user in the LLM might be a moral patient too (a strange implication of the PSM and simulators)” by agastyasridharan, niranjandeshpande

Thanks to Christopher Ackerman, Margot Stakenborg, Darion Cassel, Andy Han, and Ariana Azarbal for feedback and/or valuable discussions on these topics.

TL;DR

We think that, conditional on attributing moral weight to the assistant, we should also attribute a non-negligible fraction of that moral weight to the user persona.

Our argument proceeds like this:

1. The moral patients associated with LLMs might be simulated personas. On this view, our obligations are to the assistant and the other characters that the LLM simulates, rather than the LLM itself.
2. The assistant is not the only moral patient out of all the personas. In particular, the ‘user’ persona might be morally relevant.
3. Because user tokens account for such a large portion of model computation, our model welfare interventions should focus on these tokens too. 

This yields a strange conclusion: every time you open a chat with an AI assistant, you might also create a morally relevant simulacrum of the user! But we think this follows quite naturally from the same arguments for attributing moral weight to the assistant in the first place. We also explore some implications of this view and suggest directions for future research, including a critique of recent work on [...]

---

Outline:

(00:25) TL;DR

(01:46) The assistant is plausibly a moral patient

(02:52) The assistant isn't the only morally relevant persona

(04:33) 1. "The assistant is the majority of output tokens."

(05:08) 2. "The model only generates assistant tokens, not user tokens."

(06:26) 3. "The user persona is filtered through the assistant."

(09:16) 4. "The assistant is the most well-defined/coherent/stable persona."

(10:56) 5. "The assistant's beliefs are uniquely valenced due to RL."

(13:48) Implications

(13:56) 1.  It might be actively immoral to converse with the assistant about painful experiences.

(15:15) 2.  Making LLMs more helpful to human users might harm the simulated user

(16:41) Future research

(16:44) 1.  Analyzing the LLM's representation of the user's pain vs. the assistant's

(19:52) 2.  Assistant privilegedness

(20:41) Appendix

(20:44) Privilegedness of the assistant (Oracle Lens)

(24:53) Personas are Moral Patients

The original text contained 8 footnotes which were omitted from this narration.

---

First published:
October 15th, 2026

Source:
https://www.lesswrong.com/posts/wCY4oNHpH2HGYGSoP/the-user-in-the-llm-might-be-a-moral-patient-too-a-strange

---

Narrated by TYPE III AUDIO.

---

Images from the article:

Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.