---
theme: seriph
title: "Lesson 31 — How Can a Model Learn to Use Tools with Fewer Samples?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 7 · Model Post-Training
# How Can a Model Learn to Use Tools with Fewer Samples?
Tool-call RL, sandbox feedback, distillation, and practical boundaries
Lesson 31 of 42 · 17 minutes · RL for Learning Tool Calling; Sample Efficiency; On-Policy Distillation; Practical Tips
---
layout: center
class: text-center
---
The central question
How do we teach a model a policy over tools when most trajectories fail and every rollout is expensive?
---
# Why this problem matters
Combinatorics
Tool, argument, order, and stopping choices multiply quickly.
Grounding
Sandbox and environment feedback reveal what language alone cannot.
Dense teaching
Distillation supplies token-level targets on the student's own paths.
---
# Three ideas to keep in view
ReTool
Interleave text, code calls, and sandbox observations
On-policy distillation
Teacher scores the student's current trajectories token by token
Self-distillation
Privileged context turns the same model into a teacher
---
# The book's visual model
Tool-calling reinforcement learning reward loop
---
# Scalar RL signal vs. Dense distillation
Scalar RL signal
- One reward per trajectory
- Many samples wasted
- Can discover beyond teacher
Dense distillation
- Signal at each token
- Higher sample efficiency
- Teacher or privileged context required
Both methods need trajectories shaped like real Agent interaction.
---
# Learn on the student's own trajectory
~~~python
trajectory = student.rollout(task, tools)
teacher_logits = teacher.score(trajectory, privileged_info)
student_logits = student.score(trajectory)
loss = token_kl(student_logits, teacher_logits)
update(student, loss)
~~~
---
# Test the claim
7-9 preflight2 min
Check whether a CoT student-training run is actually ready
Observe: Data hash, sample count, dependencies, trainer compatibility, and CUDA readiness
Demo budget: 2 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ cd chapter7/cot-distillation && python train_student.py --preflight --train-data data/sft_cot_distill_aime_kimi_k3.jsonl --preflight-output validation/course-preflight.json
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Tool learning needs the same observation/action protocol at training and deployment.
Finding 2
Dense teacher distributions can succeed where sparse RL stalls.
Finding 3
A preflight separates a configured recipe from a completed training claim.
---
layout: center
---
Where the claim stops
# Boundary condition
Distillation cannot exceed information available to the teacher or privileged context, and GPU readiness is not training evidence.
---
layout: center
---
Engineering takeaway
# Design rule
Match rollout shape to deployment, preserve tool feedback, and choose the densest trustworthy signal available.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
Could privileged information make your current model a useful teacher for itself?
---
layout: center
class: text-center
---
Chapter 7 complete · Next · Lesson 32
Move learning beyond model weights into knowledge, instructions, programs, and the updater itself.
→
