---
theme: seriph
title: "Lesson 28 — How Do Preferences Become a Trainable Signal?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 7 · Model Post-Training
# How Do Preferences Become a Trainable Signal?
RLHF, reward models, KL constraints, PPO, GRPO, and DPO
Lesson 28 of 42 · 17 minutes · RLHF: From Human Preferences to Reward Models; Comparison of RL Algorithms
---
layout: center
class: text-center
---
The central question
How can human comparisons change a policy without letting optimization destroy useful behavior?
---
# Why this problem matters
Preference data
Humans compare outputs more reliably than they author perfect ones.
Reward model
Generalizes pairwise labels into a scalar training signal.
Reference policy
KL pressure limits drift away from known behavior.
---
# Three ideas to keep in view
PPO
Actor + critic + clipped policy update
GRPO
Normalize rewards within a sampled response group
DPO
Optimize chosen over rejected responses without a rollout loop
---
# The book's visual model
Group Relative Policy Optimization flow
---
# Outcome optimization vs. Preference optimization
Outcome optimization
- Can discover new outputs
- Requires rollouts
- Reward hacking risk
Preference optimization
- Uses chosen/rejected pairs
- Simpler pipeline
- Bounded by offline data
The algorithm changes how the signal is used—not whether the signal is valid.
---
# A relative advantage removes the critic
~~~python
rewards = verifier(samples)
adv = (rewards - rewards.mean()) / (rewards.std() + 1e-6)
ratio = policy.prob(samples) / old_policy.prob(samples)
loss = clipped_policy_loss(ratio, adv)
loss += beta * kl(policy, reference)
~~~
---
# Test the claim
RL evaluation check2 min
Run answer-extraction tests for an RL-trained reasoner
Observe: Whether the evaluator recognizes the trained model's answer format without inflating accuracy
Demo budget: 2 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ python -m pytest chapter7/Intuitor/tests -q
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Preference labels can train either an explicit reward model or a direct objective.
Finding 2
Relative rewards reduce value-model complexity but do not fix a bad verifier.
Finding 3
KL is a steering constraint, not proof that useful capabilities are retained.
---
layout: center
---
Where the claim stops
# Boundary condition
Offline preference methods cannot explore behaviors absent from their comparison data.
---
layout: center
---
Engineering takeaway
# Design rule
Choose the simplest optimizer that can use your signal, then spend most of the effort validating the signal and holdout behavior.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
What shortcut could maximize your proposed reward while making the real product worse?
---
layout: center
class: text-center
---
Next · Lesson 29
Move attention from algorithm names to the data and environment that define the signal.
→
