---
theme: seriph
title: "Lesson 27 — When Should You Teach with Examples—and When with Rewards?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 7 · Model Post-Training
# When Should You Teach with Examples—and When with Rewards?
SFT, loss masking, distribution shift, and the form-first rule
Lesson 27 of 42 · 19 minutes · SFT; When to Choose SFT and When to Choose RL; Single-Turn Reinforcement Learning
---
layout: center
class: text-center
---
The central question
Is the target capability a stable mapping to imitate or a strategy that must survive new situations?
---
# Why this problem matters
SFT
Dense token-level supervision; stable and sample-efficient.
RL
Sparse trajectory feedback; costly but allows exploration.
Shift
Test whether the learned behavior survives changed rules and inputs.
---
# Three ideas to keep in view
Loss masking
Supervise the assistant response—not the user prompt
Form first
Stabilize parsable output before optimizing strategy
Holdout shift
Change values or environments while preserving the rule
---
# The book's visual model
SFT followed by RL as a two-stage training pipeline
---
# Use SFT vs. Consider RL
Use SFT
- Known demonstrations
- Format/style/protocol
- Deployment matches training
Consider RL
- Outcome can be verified
- Many valid strategies
- Generalization under shift matters
SFT and RL are sequential tools—not rival ideologies.
---
# SFT masks the prompt tokens
~~~python
tokens = prompt_ids + response_ids
labels = [-100] * len(prompt_ids) + response_ids
loss = cross_entropy(model(tokens), labels)
loss.backward()
# only the demonstrated response is supervised
~~~
---
# Test the claim
7-4 evidence2 min
Audit retained VLM pre-training and SFT evidence
Observe: Hashed outputs, blind judgments, matched configurations, and negative results
7-5 evidence2 min
Audit continued-pretraining trade-offs
Observe: New-language gain, retained English ability, and persistent factual errors
Demo budget: 4 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ python chapter7/MiniMind-pretrain/validation/validate_vlm_evidence.py
$ python chapter7/continued-pretraining/validation/validate_evidence.py
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
SFT efficiently learns explicit protocols represented in examples.
Finding 2
RL is justified when a verifier can reward strategies beyond one reference answer.
Finding 3
Training gains must be tested beside retention and distribution-shift failures.
---
layout: center
---
Where the claim stops
# Boundary condition
The slogan 'SFT memorizes, RL generalizes' is a tendency under controlled conditions—not a guarantee for every model and task.
---
layout: center
---
Engineering takeaway
# Design rule
Use SFT until output is stable; add RL only when exploration and verified generalization justify its cost.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
What deployment change would reveal that your fine-tuned model learned an example instead of a rule?
---
layout: center
class: text-center
---
Next · Lesson 28
Translate preferences and outcomes into optimization signals without losing the base policy.
→