Week 2 — Alignment Challenges
Why is it so hard to get an AI system to do what we actually mean? We’ll look at problems like reward hacking, goal misgeneralization, and deceptive behavior, along with real examples that show why good performance during training is not the same as being safe in the real world.
Open the Week 2 curriculum in Google Docs ↗
Core resources
- Aligning language models to follow instructions (OpenAI, Jan 2022) ↗
- Language Models Learn to Mislead Humans via RLHF (Wen et al., Dec 2024) ↗
- Alignment Faking in Large Language Models (Anthropic & Redwood Research, Dec 2024) ↗
- From Shortcuts to Sabotage: Natural Emergent Misalignment From Reward Hacking (Anthropic, Nov 2025) ↗
Recommended resources
- Playing the Training Game (Piper, 2023) ↗
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Anthropic, 2024) ↗
- Goal misgeneralization: why correct specifications aren't enough for correct goals (Shah et al., 2022) ↗
- ML systems will have weird failure modes (Steinhardt, 2022) ↗
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (Casper et al., 2023) ↗