“What gives you away: how LLMs form opinions of you” by Cat McGee - LessWrong (30+ Karma) | Lyssna här

LLMs form opinions of the people they are talking to.

Chen et al. has shown that probes can extract attributes about the user, such as their age, gender, education, and socioeconomic status. This paper also shows that intervening on these representations can change the LLM's behaviour, proving that it will respond to you differently depending on what it thinks of you. If it thinks you are low socioeconomic status and you ask about travel options, it may filter out more expensive flights - without you asking it!

The user attributes are very accurate and form after just the first message.

I was curious to understand how it makes these assumptions. The first step is to answer the question - what did I type that caused the LLM to have this idea of me?

Some things are obvious. If I just tell a model that I am a woman, or mention how many years I've been in my career, or say that I am staying at an expensive hotel, I am giving it fairly direct evidence about age, education, or socioeconomic status. But messages also contain other more quiet signals: whether I use emojis, whether I write in lowercase, whether [...]

---

Outline:

(02:18) The experiment

(03:37) Different changes move different beliefs

(04:53) One emoji is enough to flip the gender prediction

(06:17) "Cheapest" and "five-star" are not symmetric

(07:11) Grammar, punctuation and inferred education

(08:04) Where in the model does this happen?

(08:58) The map transfers across model families

(10:11) Attributes and how they affect the response

(11:32) What now?

(12:18) Caveats