ai-agent-book/slides/lesson-26.md

theme, title, info, author, transition, mdc, lineNumbers, monaco, aspectRatio, canvasWidth, layout, class

theme	title	info	author	transition	mdc	lineNumbers	monaco	aspectRatio	canvasWidth	layout	class
seriph	Lesson 26 — Why Does Model Training Happen in Three Stages?	English video course for AI Agents in Depth	Bojie Li	slide-left	true	false	false	16/9	980	cover	cover

Improve · Chapter 7 · Model Post-Training

Why Does Model Training Happen in Three Stages?

Pre-training, SFT, RL, and the agent-environment loop

Lesson 26 of 42 · 17 minutes · Pre-training, SFT, RL: A Three-Stage Panorama; Classic RL Agents; Pre-training Basics

---

Improve · Chapter 7 · Model Post-Training

Problems this chapter will solve

Lesson 26

Why Does Model Training Happen in Three Stages?

Lesson 27

When Should You Teach with Examples—and When with Rewards?

Lesson 28

How Do Preferences Become a Trainable Signal?

Lesson 29

Why Do Data and Environments Matter More Than the Algorithm?

Lesson 30

How Do You Reward a Long Agent Trajectory?

Lesson 31

How Can a Model Learn to Use Tools with Fewer Samples?

---

Why this problem matters

Pre-training

Acquire language, priors, knowledge, and basic reasoning.

SFT

Learn the response and tool-use protocol from demonstrations.

RL

Explore decisions and increase actions that earn reward.

---

Three ideas to keep in view

Policy

A probability distribution over the next action

Environment

The world that returns a new state and reward

Update

Move probability toward behavior supported by the signal

---

The book's visual model

Q-learning and LLM Agent architectures in a treasure hunt

---

Classic RL vs. LLM Agent

Classic RL

• Small state/action space
• Learns mainly from trials
• Explicit value estimates

LLM Agent

• Language observations/actions
• Strong pretrained priors
• Reasons in context before acting

The loop is shared; the representation and prior knowledge are radically different.

---

The Q-learning update

target = reward + gamma * max(Q[next_state])
error = target - Q[state, action]
Q[state, action] += alpha * error
state = next_state
# experience changes the next choice

---

Test the claim

7-12 min

Watch Q-learning discover hidden game mechanics

Observe: Learning curve, exploration decay, and final greedy success

Demo budget: 2 minutes · one contiguous terminal block

---

class: course-terminal

Live demo

Switching to the terminal

$ cd chapter1/learning-from-experience && python experiment.py --mode qlearning --rl-episodes 10000 --eval-episodes 100 --seed 42

Run the command(s), narrate decisions, and point to the observation—not just the output.

---

What the evidence supports

Finding 1

Pre-training supplies priors that tabular RL must discover from scratch.

Finding 2

SFT and pre-training both predict tokens; their data and loss masks differ.

Finding 3

RL needs an environment capable of producing a meaningful signal.

---

layout: center

Where the claim stops

Boundary condition

The treasure hunt clarifies the loop, but its small state space does not represent the scale of language-model training.

---

layout: center

Engineering takeaway

Design rule

Before choosing a training method, name the capability, the representation it should change, and the signal available to teach it.

---

Continue the experiment

Experiment 7-2: Q-learning vs. LLM Agent chapter1/learning-from-experience/ Q-learning grid world book-en/images/fig7-3.svg Classic vs. modern Agent book-en/images/fig7-5.svg Training paradigm evolution book-en/images/fig7-6.svg

---

layout: center class: text-center

Pause and apply

Your turn

Which capability in your Agent comes from weights, and which is reconstructed from context every run?

---

layout: center class: text-center

Next · Lesson 27

Decide when demonstrations are enough and when exploration is worth the cost.

→