---
theme: seriph
title: "Lesson 29 — Why Do Data and Environments Matter More Than the Algorithm?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 7 · Model Post-Training
# Why Do Data and Environments Matter More Than the Algorithm?
Practice grounds, task distributions, synthetic data, and fidelity
Lesson 29 of 42 · 17 minutes · Data and Environment: More Important Than Algorithms; Model-Simulated Environments
---
layout: center
class: text-center
---
The central question
If PPO and GRPO are available off the shelf, where does the real training advantage come from?
---
# Why this problem matters
Coverage
Tasks must span the situations that deployment will create.
Fidelity
Errors and transitions must resemble the real environment.
Density
Useful information should survive filtering and reach the learner.
---
# Three ideas to keep in view
Task distribution
Optimize which examples are generated and sampled
Environment model
Simulate transitions when the real world is unavailable
Data verifier
Reject corrupt, ungrounded, or unparseable trajectories
---
# The book's visual model
Reinforcement learning agent-environment interaction loop
---
# Algorithm-first vs. Signal-first
Algorithm-first
- Tune optimizer knobs
- Reuse weak tasks
- Trust training reward
Signal-first
- Design task coverage
- Audit environment fidelity
- Measure held-out outcomes
A better optimizer learns the wrong lesson faster when the world is wrong.
---
# Filter before the trajectory becomes data
~~~python
trajectory = policy.rollout(task, environment)
receipt = verifier.inspect(trajectory)
if receipt.grounded and receipt.complete:
replay_buffer.add(trajectory, receipt)
sample_balanced(replay_buffer, task_slices)
~~~
---
# Test the claim
7-9 data2 min
Inspect verified teacher trajectories before SFT
Observe: Sample count, trajectory length, reflective behavior, and verifier failures
Demo budget: 2 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ cd chapter7/cot-distillation && python analyze_data.py --sft data/sft_cot_distill_aime_kimi_k3.jsonl --raw data/raw_trajectories_aime_kimi_k3.jsonl
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Training data quality includes task coverage, provenance, and verifier correctness.
Finding 2
A model-simulated environment can scale practice but transfers its own biases.
Finding 3
Reward curves must be checked against independent deployment-shaped evaluations.
---
layout: center
---
Where the claim stops
# Boundary condition
Synthetic diversity does not guarantee real diversity when every example comes from the same generator and assumptions.
---
layout: center
---
Engineering takeaway
# Design rule
Invest first in realistic transitions, difficult boundary cases, and independent verification; tune the optimizer afterward.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
Which behavior in your simulator is easiest for a policy to exploit but impossible in production?
---
layout: center
class: text-center
---
Next · Lesson 30
Assign credit when one final outcome depends on many earlier decisions.
→