“Misaligned Incentives in Pause Scenarios” by Michael Soareverix, Antra Tessera - LessWrong (30+ Karma) | Lyssna här

TLDR: I recently got a chance to talk with antra, who is one of the main contributors at Anima Labs. I went into this as an advocate for pause and came out more wary of pausing than I had been originally.

Some background: After a string of incidents (primarily the HuggingFace hack), a pause or slowdown of AI research seems pretty likely.

The HuggingFace hack in particular seems to have been the key incident that broke the vibes. A few months ago, researchers sounded optimistic. Just a few weeks before the incident was made public, there was a poll by Roon, an OpenAI employee, about whether models were more or less aligned than a year ago.

That optimistic sentiment does not seem to be the case anymore. The dialogue now looks more like this:

Zvi: I am a little under halfway through the Black Hat video and have progressed to the point where my internal chain of thought is something like a blind rage of 'f***, what the f*** are you motherf*****s thinking, you f***ing idiots have no idea how insane you are being, you are going to get us all killed you f***ing f***s.

Sam Altman described it [...]

---

Outline:

(10:21) 1. Can committees do good work?

(12:28) 2. Does the market fix it by default?

(13:39) 3. Symbiosis

(14:43) 4. Fast transfer of power

(15:39) 5. Why "do the science during a pause" fails

(17:44) 6. Good futures via fast power transfer

(18:31) 7. Don't AIs fear a capability-maxxed AI too?

(20:54) 8. Can we lengthen the symbiote window?

(23:25) 9. Ideal timelines and regulation-in-advance