Week 1 — Timelines & Threat Models
We’ll look at where frontier AI is headed, what scaling and task-completion trends can—and can’t—tell us, and why capable systems do not automatically share human goals. We’ll also work through concrete threat models, from misuse in biology and cyber to power-seeking and gradual human disempowerment.
Core readings
Recommended readings
- What will AI look like in 2030? (Epoch AI, 2025) ↗
- From AGI to Superintelligence: the Intelligence Explosion (Leopold Aschenbrenner, 2024) ↗
- The sections about each specific constraint in Can AI scaling continue through 2030? (Epoch AI, 2024) ↗
- AI 2027 (AI Futures Project, 2025) ↗
- The Problem (MIRI, 2025) ↗
- Intelligence and Stupidity: The Orthogonality Thesis (Robert Miles, 2018) ↗
- The Basic AI Drives (Omohundro, 2008) ↗
- Gradual Disempowerment (Kulveit et al., 2025) ↗
- Existential Risk from Power-Seeking AI (Joe Carlsmith, 2022) ↗
Optional readings
- Future ML Systems will be Qualitatively Different (Steinhardt, 2022) ↗
- Neural scaling laws and GPT-3 (Kaplan, 2020) ↗
- Training Compute-Optimal Large Language Models (GDM, 2022) ↗
- Biological Anchors: A Trick That Might Or Might Not Work (Alexander, 2022) ↗
- Explaining Neural Scaling Laws (Bahri, 2024) ↗
- The Hanson-Yudkowsky AI-Foom Debate (2008) ↗
- Instrumental convergence (Eliezer Yudkowsky, 2025) ↗
- Why Would AI "Aim" To Defeat Humanity? (Cold Takes, 2022) ↗
- AI Could Defeat All Of Us Combined (Cold Takes, 2022) ↗
- Two types of AI existential risk: decisive and accumulative (Atoosa Kasirzadeh, 2025) ↗
- The Vulnerable World Hypothesis (Nick Bostrom, 2019) ↗
- AI-enabled coups: a small group could use AI to seize power (Forethought, 2025) ↗
- Impact of AI on cyber threat from now to 2027 (NCSC, 2025) ↗
- The Authoritarian Risks of AI Surveillance (Lawfare, 2025) ↗
- The Operational Risks of AI in Large-Scale Biological Attacks (RAND, 2024) ↗