---
theme: seriph
title: "Lesson 30 — How Do You Reward a Long Agent Trajectory?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 7 · Model Post-Training
# How Do You Reward a Long Agent Trajectory?
Credit assignment, reward density, process signals, and path penalties
Lesson 30 of 42 · 17 minutes · From Single-Turn to Multi-Turn; Credit Assignment; Process vs. Outcome Reward; RLVP
---
layout: center
class: text-center
---
The central question
When the final result fails, which earlier tool choice should the model change?
---
# Why this problem matters
Sparse outcome
One final bit leaves most actions unexplained.
Process evidence
Tool errors and rule violations identify local mistakes.
Partial credit
Reachable progress can rescue information from all-fail groups.
---
# Three ideas to keep in view
Outcome reward
Score the completed task—not a convenient proxy
Process reward
Evaluate intermediate reasoning or actions
RLVP
Reward the outcome; penalize verified path violations
---
# The book's visual model
Credit assignment across a multi-turn interaction
---
# Outcome only vs. Outcome + path evidence
Outcome only
- Objective final target
- Simple verifier
- Very sparse credit
Outcome + path evidence
- Retains final goal
- Uses observed violations
- Denser diagnosis
Path signals should constrain the route without replacing the destination.
---
# Separate success from path violations
~~~python
outcome = task_verifier(final_state)
violations = rule_verifier(trajectory)
progress = reachable_subgoals(trajectory)
reward = outcome - penalty(violations)
reward += partial_credit(progress, only_if_all_fail=True)
~~~
---
# Test the claim
7-14 gates2 min
Run verifier regressions for trajectory data
Observe: Malformed samples rejected before they can become supervision
Demo budget: 2 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ python -m pytest chapter7/cot-distillation/test_student_pipeline.py chapter7/cot-distillation/test_empty_problems.py -q
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Multi-turn tasks turn one outcome into a temporal attribution problem.
Finding 2
Environment feedback contains information that scalar outcome rewards discard.
Finding 3
A process metric becomes dangerous when it is easier to optimize than the real goal.
---
layout: center
---
Where the claim stops
# Boundary condition
An LLM process judge can reward plausible-looking steps that did not causally produce the outcome.
---
layout: center
---
Engineering takeaway
# Design rule
Keep an objective outcome gate, add only externally verified path signals, and test explicitly for reward hacking.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
Which intermediate signal in your Agent is evidence of progress, and which is merely correlated with it?
---
layout: center
class: text-center
---
Next · Lesson 31
Apply these signals to the combinatorial problem of learning when and how to call tools.
→
