In his video he went over LLM foundation model training, supervised finetuning, and the latest exploratory RL and RLHF training. And there is a section where he explains why DeepSeek R1 paper is a big deal (at a high level, yet super helpful). I feel like I have a much better understanding of how RL is helping LLM post-training and how RLHF is (and is not) really RL and its shortcomings.

I watched the whole video (3.5hours @ 2x) and would recommend it anyone who’s interested in LLM or working with LLM since it will help you understand its quirks.

Note, I watched many of Andrej’s previous videos. For example, the one on tokenization (which explains why asking GPT to output YAML instead of JSON saves more tokens, among other things.) He is a great educator.