ai-agent-book/slides/lesson-29.md

theme, title, info, author, transition, mdc, lineNumbers, monaco, aspectRatio, canvasWidth, layout, class

theme	title	info	author	transition	mdc	lineNumbers	monaco	aspectRatio	canvasWidth	layout	class
seriph	Lesson 29 — Why Do Data and Environments Matter More Than the Algorithm?	English video course for AI Agents in Depth	Bojie Li	slide-left	true	false	false	16/9	980	cover	cover

Improve · Chapter 7 · Model Post-Training

Why Do Data and Environments Matter More Than the Algorithm?

Practice grounds, task distributions, synthetic data, and fidelity

Lesson 29 of 42 · 17 minutes · Data and Environment: More Important Than Algorithms; Model-Simulated Environments

---

layout: center class: text-center

The central question

If PPO and GRPO are available off the shelf, where does the real training advantage come from?

---

Why this problem matters

Coverage

Tasks must span the situations that deployment will create.

Fidelity

Errors and transitions must resemble the real environment.

Density

Useful information should survive filtering and reach the learner.

---

Three ideas to keep in view

Task distribution

Optimize which examples are generated and sampled

Environment model

Simulate transitions when the real world is unavailable

Data verifier

Reject corrupt, ungrounded, or unparseable trajectories

---

The book's visual model

Reinforcement learning agent-environment interaction loop

---

Algorithm-first vs. Signal-first

Algorithm-first

• Tune optimizer knobs
• Reuse weak tasks
• Trust training reward

Signal-first

• Design task coverage
• Audit environment fidelity
• Measure held-out outcomes

A better optimizer learns the wrong lesson faster when the world is wrong.

---

Filter before the trajectory becomes data

trajectory = policy.rollout(task, environment)
receipt = verifier.inspect(trajectory)
if receipt.grounded and receipt.complete:
    replay_buffer.add(trajectory, receipt)
sample_balanced(replay_buffer, task_slices)

---

Test the claim

7-9 data2 min

Inspect verified teacher trajectories before SFT

Observe: Sample count, trajectory length, reflective behavior, and verifier failures

Demo budget: 2 minutes · one contiguous terminal block

---

class: course-terminal

Live demo

Switching to the terminal

$ cd chapter7/cot-distillation && python analyze_data.py --sft data/sft_cot_distill_aime_kimi_k3.jsonl --raw data/raw_trajectories_aime_kimi_k3.jsonl

Run the command(s), narrate decisions, and point to the observation—not just the output.

---

What the evidence supports

Finding 1

Training data quality includes task coverage, provenance, and verifier correctness.

Finding 2

A model-simulated environment can scale practice but transfers its own biases.

Finding 3

Reward curves must be checked against independent deployment-shaped evaluations.

---

layout: center

Where the claim stops

Boundary condition

Synthetic diversity does not guarantee real diversity when every example comes from the same generator and assumptions.

---

layout: center

Engineering takeaway

Design rule

Invest first in realistic transitions, difficult boundary cases, and independent verification; tune the optimizer afterward.

---

Continue the experiment

Experiment 7-8: prompt distillation chapter8/prompt-distillation/ Experiment 7-9: CoT distillation chapter7/cot-distillation/ Experiment 7-10: adaptive reasoning length chapter7/AdaptThink/ Autodata and simulated-environment discussion book-en/chapter7.md

---

layout: center class: text-center

Pause and apply

Your turn

Which behavior in your simulator is easiest for a policy to exploit but impossible in production?

---

layout: center class: text-center

Next · Lesson 30

Assign credit when one final outcome depends on many earlier decisions.

→