Week 2 — Alignment Challenges
Why is it so hard to get an AI system to do what we actually mean? We’ll look at problems like reward hacking, goal misgeneralization, and deceptive behavior, along with real examples that show why good performance during training is not the same as being safe in the real world.
Core readings
- Aligning language models to follow instructions (OpenAI, Jan 2022) ↗
- Language Models Learn to Mislead Humans via RLHF (Wen et al., Dec 2024) ↗
- Alignment Faking in Large Language Models (Anthropic & Redwood Research, Dec 2024) ↗
- From Shortcuts to Sabotage: Natural Emergent Misalignment From Reward Hacking (Anthropic, Nov 2025) ↗
Recommended readings
- Large Language Models can Strategically Deceive their Users when Put Under Pressure (Apollo Research, 2023) ↗
- The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment (Robert Miles, 2021) ↗
- Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (Anthropic, 2024) ↗
- Constitutional AI: Harmlessness from AI Feedback (Anthropic, 2022) ↗
- Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback (Casper et al., 2023) ↗
- ML systems will have weird failure modes (Steinhardt, 2022) ↗
- Sycophancy to subterfuge: Investigating reward tampering in language models (Anthropic, 2024) ↗
- Frontier Models are Capable of In-Context Scheming (Apollo Research, 2024) ↗
- Distillation of "How likely is deceptive alignment?" (originally Evan Hubinger, 2022) ↗
Optional readings
- Deep RL from human preferences: blog post (Christiano et al., 2017) ↗
- Playing the Training Game (Piper, 2023) ↗
- Sycophancy to subterfuge: Investigating reward tampering in language models (Anthropic, 2024) ↗
- Scaling Laws for Reward Model Overoptimization (OpenAI, 2022) ↗
- Concrete Problems in AI Safety (Amodei et al., 2016) ↗
- Goal misgeneralization: why correct specifications aren't enough for correct goals (Shah et al., 2022) ↗
- The alignment problem from a deep learning perspective (Ngo et al., 2022) ↗
- Goal Misgeneralization in Deep Reinforcement Learning (Langosco et al., 2022) ↗
- Optimal policies tend to seek power: NeurIPS spotlight presentation (Turner et al., 2022) ↗
- Language Models as Agent Models (Andreas, 2022) ↗