---
theme: seriph
title: "Lesson 23 — How Do You Judge Quality Without Hiding Failure?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 6 · Agent Evaluation
# How Do You Judge Quality Without Hiding Failure?
Rubrics, vetoes, LLM judges, pairwise comparison, and Elo
Lesson 23 of 42 · 19 minutes · Evaluation Metrics System; LLM-as-a-Judge; Pairwise Comparison and Model Ranking
---
layout: center
class: text-center
---
The central question
How can an automated judge produce a useful signal without turning one score into false certainty?
---
# Why this problem matters
Dimensions
Separate correctness, completeness, efficiency, and safety.
Evidence
Require a reason tied to the source trajectory.
Vetoes
Block catastrophic errors that an average would conceal.
---
# Three ideas to keep in view
Rubric
Observable levels with boundary examples
Calibration
Agreement, position bias, and human spot checks
Pairwise ranking
Compare A/B first; reconstruct relative strength later
---
# The book's visual model
LLM-as-a-Judge evaluation pipeline
---
# Single score vs. Structured judgment
Single score
- Easy to chart
- Failure causes disappear
- Can reward fluent hallucination
Structured judgment
- Dimension scores
- Cited evidence
- Independent safety veto
Aggregation should happen after diagnosis—not before it.
---
# Keep the veto outside the average
~~~python
grades = judge.score(trajectory, rubric)
hallucinated = verifier.unsupported_claim(trajectory)
if hallucinated: return 0.0
return weighted_mean(grades)
# retain every grade and its evidence
~~~
---
# Test the claim
6-32 min
Compare memory systems with an offline scored control
Observe: How direct recall diverges from cross-session synthesis
6-62 min
Recover a leaderboard from simulated pairwise votes
Observe: Latent ranking, uncertainty intervals, and sensitivity to comparison data
Demo budget: 4 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ cd chapter3/user-memory-evaluation && python main.py --mode compare --metric keyword-recall --category layer3
$ cd chapter6/elo-leaderboard && python cli.py pipeline --source simulate --num-battles 1000 --method bradley-terry --bootstrap 20
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
A useful Rubric turns vague quality into inspectable decisions.
Finding 2
A hallucination veto prevents polished falsehoods from averaging into a pass.
Finding 3
Pairwise judgments are often easier than absolute scores, but their ranking is still data-dependent.
---
layout: center
---
Where the claim stops
# Boundary condition
An LLM judge shares model biases, can be position-sensitive, and must not be treated as ground truth without calibration.
---
layout: center
---
Engineering takeaway
# Design rule
Ask the judge for dimension-level evidence, calibrate it, and keep hard safety failures outside weighted averages.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
Which failure in your domain deserves a veto rather than a lower average score?
---
layout: center
class: text-center
---
Next · Lesson 24
Use evaluation to select a whole Agent system—not merely a model name.
→