ai-agent-book/slides/lesson-31.md

theme, title, info, author, transition, mdc, lineNumbers, monaco, aspectRatio, canvasWidth, layout, class

theme	title	info	author	transition	mdc	lineNumbers	monaco	aspectRatio	canvasWidth	layout	class
seriph	Lesson 31 — How Can a Model Learn to Use Tools with Fewer Samples?	English video course for AI Agents in Depth	Bojie Li	slide-left	true	false	false	16/9	980	cover	cover

Improve · Chapter 7 · Model Post-Training

How Can a Model Learn to Use Tools with Fewer Samples?

Tool-call RL, sandbox feedback, distillation, and practical boundaries

Lesson 31 of 42 · 17 minutes · RL for Learning Tool Calling; Sample Efficiency; On-Policy Distillation; Practical Tips

---

layout: center class: text-center

The central question

How do we teach a model a policy over tools when most trajectories fail and every rollout is expensive?

---

Why this problem matters

Combinatorics

Tool, argument, order, and stopping choices multiply quickly.

Grounding

Sandbox and environment feedback reveal what language alone cannot.

Dense teaching

Distillation supplies token-level targets on the student's own paths.

---

Three ideas to keep in view

ReTool

Interleave text, code calls, and sandbox observations

On-policy distillation

Teacher scores the student's current trajectories token by token

Self-distillation

Privileged context turns the same model into a teacher

---

The book's visual model

Tool-calling reinforcement learning reward loop

---

Scalar RL signal vs. Dense distillation

Scalar RL signal

• One reward per trajectory
• Many samples wasted
• Can discover beyond teacher

Dense distillation

• Signal at each token
• Higher sample efficiency
• Teacher or privileged context required

Both methods need trajectories shaped like real Agent interaction.

---

Learn on the student's own trajectory

trajectory = student.rollout(task, tools)
teacher_logits = teacher.score(trajectory, privileged_info)
student_logits = student.score(trajectory)
loss = token_kl(student_logits, teacher_logits)
update(student, loss)

---

Test the claim

7-9 preflight2 min

Check whether a CoT student-training run is actually ready

Observe: Data hash, sample count, dependencies, trainer compatibility, and CUDA readiness

Demo budget: 2 minutes · one contiguous terminal block

---

class: course-terminal

Live demo

Switching to the terminal

$ cd chapter7/cot-distillation && python train_student.py --preflight --train-data data/sft_cot_distill_aime_kimi_k3.jsonl --preflight-output validation/course-preflight.json

Run the command(s), narrate decisions, and point to the observation—not just the output.

---

What the evidence supports

Finding 1

Tool learning needs the same observation/action protocol at training and deployment.

Finding 2

Dense teacher distributions can succeed where sparse RL stalls.

Finding 3

A preflight separates a configured recipe from a completed training claim.

---

layout: center

Where the claim stops

Boundary condition

Distillation cannot exceed information available to the teacher or privileged context, and GPU readiness is not training evidence.

---

layout: center

Engineering takeaway

Design rule

Match rollout shape to deployment, preserve tool feedback, and choose the densest trustworthy signal available.

---

Continue the experiment

Experiment 7-15: ReTool chapter7/retool/ Experiment 7-16: AWorld training chapter7/AWorld-train/ AWorld source checkout chapter7/AWorld/ On-policy distillation discussion book-en/chapter7.md Tool ecosystem architecture book-en/images/fig7-18.svg

---

layout: center class: text-center

Pause and apply

Your turn

Could privileged information make your current model a useful teacher for itself?

---

layout: center class: text-center

Chapter 7 complete · Next · Lesson 32

Move learning beyond model weights into knowledge, instructions, programs, and the updater itself.

→