---
theme: seriph
title: "Lesson 26 — Why Does Model Training Happen in Three Stages?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 7 · Model Post-Training
# Why Does Model Training Happen in Three Stages?
Pre-training, SFT, RL, and the agent-environment loop
Lesson 26 of 42 · 17 minutes · Pre-training, SFT, RL: A Three-Stage Panorama; Classic RL Agents; Pre-training Basics
---
Improve · Chapter 7 · Model Post-Training
# Problems this chapter will solve
Lesson 26
Why Does Model Training Happen in Three Stages?
Lesson 27
When Should You Teach with Examples—and When with Rewards?
Lesson 28
How Do Preferences Become a Trainable Signal?
Lesson 29
Why Do Data and Environments Matter More Than the Algorithm?
Lesson 30
How Do You Reward a Long Agent Trajectory?
Lesson 31
How Can a Model Learn to Use Tools with Fewer Samples?
---
# Why this problem matters
Pre-training
Acquire language, priors, knowledge, and basic reasoning.
SFT
Learn the response and tool-use protocol from demonstrations.
RL
Explore decisions and increase actions that earn reward.
---
# Three ideas to keep in view
Policy
A probability distribution over the next action
Environment
The world that returns a new state and reward
Update
Move probability toward behavior supported by the signal
---
# The book's visual model
Q-learning and LLM Agent architectures in a treasure hunt
---
# Classic RL vs. LLM Agent
Classic RL
- Small state/action space
- Learns mainly from trials
- Explicit value estimates
LLM Agent
- Language observations/actions
- Strong pretrained priors
- Reasons in context before acting
The loop is shared; the representation and prior knowledge are radically different.
---
# The Q-learning update
~~~python
target = reward + gamma * max(Q[next_state])
error = target - Q[state, action]
Q[state, action] += alpha * error
state = next_state
# experience changes the next choice
~~~
---
# Test the claim
7-12 min
Watch Q-learning discover hidden game mechanics
Observe: Learning curve, exploration decay, and final greedy success
Demo budget: 2 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ cd chapter1/learning-from-experience && python experiment.py --mode qlearning --rl-episodes 10000 --eval-episodes 100 --seed 42
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Pre-training supplies priors that tabular RL must discover from scratch.
Finding 2
SFT and pre-training both predict tokens; their data and loss masks differ.
Finding 3
RL needs an environment capable of producing a meaningful signal.
---
layout: center
---
Where the claim stops
# Boundary condition
The treasure hunt clarifies the loop, but its small state space does not represent the scale of language-model training.
---
layout: center
---
Engineering takeaway
# Design rule
Before choosing a training method, name the capability, the representation it should change, and the signal available to teach it.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
Which capability in your Agent comes from weights, and which is reconstructed from context every run?
---
layout: center
class: text-center
---
Next · Lesson 27
Decide when demonstrations are enough and when exploration is worth the cost.
→
