Ep 10 - Accelerated training to become an AI safety researcher w/ Ryan Kidd (Co-Director, MATS)

We speak with Ryan Kidd, Co-Director at ML Alignment & Theory Scholars (MATS) program, previously "SERI MATS".

MATS (https://www.matsprogram.org/) provides research mentorship, technical seminars, and connections to help new AI researchers get established and start producing impactful research towards AI safety & alignment.

Prior to MATS, Ryan completed a PhD in Physics at the University of Queensland (UQ) in Australia.

We talk about:

* What the MATS program is
* Who should apply to MATS (next *deadline*: Nov 17 midnight PT)
* Research directions being explored by MATS mentors, now and in the past
* Promising alignment research directions & ecosystem gaps , in Ryan's view

Hosted by Soroush Pour. Follow me for more AGI content:
* Twitter: https://twitter.com/soroushjp
* LinkedIn: https://www.linkedin.com/in/soroushjp/

== Show links ==

-- About Ryan --

* Twitter: https://twitter.com/ryan_kidd44
* LinkedIn: https://www.linkedin.com/in/ryan-kidd-1b0574a3/
* MATS: https://www.matsprogram.org/
* LISA: https://www.safeai.org.uk/
* Manifold: https://manifold.markets/

-- Further resources --

* Book: “The Precipice” - https://theprecipice.com/
* Ikigai - https://en.wikipedia.org/wiki/Ikigai
* Fermi paradox - https://en.wikipedia.org/wiki/Fermi_p...
* Ajeya Contra - Bioanchors - https://www.cold-takes.com/forecastin...
* Chomsky hierarchy & LLM transformers paper + external memory - https://en.wikipedia.org/wiki/Chomsky...
* AutoGPT - https://en.wikipedia.org/wiki/Auto-GPT
* BabyAGI - https://github.com/yoheinakajima/babyagi
* Unilateralist's curse - https://forum.effectivealtruism.org/t...
* Jeffrey Ladish & team - fine tuning to remove LLM safeguards - https://www.alignmentforum.org/posts/...
* Epoch AI trends - https://epochai.org/trends
* The demon "Moloch" - https://slatestarcodex.com/2014/07/30...
* AI safety fundamentals course - https://aisafetyfundamentals.com/
* Anthropic sycophancy paper - https://www.anthropic.com/index/towar...
* Promising technical alignment research directions
    * Scalable oversight
        * Recursive reward modelling - https://deepmindsafetyresearch.medium...
        * RLHF - could work for a while, but unlikely forever as we scale
    * Interpretability
        * Mechanistic interpretability
            * Paper: GPT4 labelling GPT2 - https://openai.com/research/language-...
        * Concept based interpretability
            * Rome paper - https://rome.baulab.info/
        * Developmental interpretability
            * devinterp.com - http://devinterp.com
            * Timaeus - https://timaeus.co/
        * Internal consistency
            * Colin Burns research - https://arxiv.org/abs/2212.03827
* Threat modelling / capabilities evaluation & demos
    * Paper: Can large language models democratize access to dual-use biotechnology? - https://arxiv.org/abs/2306.03809
    * ARC Evals - https://evals.alignment.org/
    * Palisade Research - https://palisaderesearch.org/
    * Paper: Situational awareness with Owain Evans - https://arxiv.org/abs/2309.00667
* Gradient hacking - https://www.lesswrong.com/posts/uXH4r6MmKPedk8rMA/gradient-hacking
* Past scholar's work
    * Apollo Research - https://www.apolloresearch.ai/
    * Leap Labs - https://www.leap-labs.com/
    * Timaeus - https://timaeus.co/
* Other orgs mentioned
    * Redwood Research - https://redwoodresearch.org/

Recorded Oct 25, 2023