---
theme: seriph
title: "Lesson 25 — Did the Agent Improve—or Did the Numbers Move?"
info: "English video course for AI Agents in Depth"
author: Bojie Li
transition: slide-left
mdc: true
lineNumbers: false
monaco: false
aspectRatio: 16/9
canvasWidth: 980
layout: cover
class: cover
---
Improve · Chapter 6 · Agent Evaluation
# Did the Agent Improve—or Did the Numbers Move?
Significance, observability, ablations, and production evaluation
Lesson 25 of 42 · 17 minutes · Statistical Significance; Agent Observability; Internal Evaluation Infrastructure; Simulation Fidelity
---
layout: center
class: text-center
---
The central question
What evidence is required before an evaluation delta becomes an engineering decision?
---
# Why this problem matters
Uncertainty
Repeated samples expose variance and paired differences.
Observability
Traces connect aggregate regressions to mechanisms.
Release control
Feature flags, A/B tests, rollback, and privacy-aware analytics
---
# Three ideas to keep in view
Paired test
Run old and new systems on the same cases
Confidence interval
Report a plausible range—not only a mean
Two-layer flags
Separate mechanism enablement from experiment assignment
---
# The book's visual model
Observability technology stack for Agent systems
---
# Scoreboard vs. Evaluation infrastructure
Scoreboard
- One aggregate
- No trace linkage
- Manual reruns
Evaluation infrastructure
- Slices + uncertainty
- Trajectory-level observability
- Repeatable release gates
The purpose of a benchmark report is to generate testable hypotheses.
---
# Pair cases before estimating the delta
~~~python
deltas = [new[c] - old[c] for c in shared_cases]
estimate, interval = bootstrap_mean(deltas)
if interval.low <= 0: hold_release()
else: canary(new_system)
monitor_slices_and_rollback()
~~~
---
# Test the claim
6-3 evidence2 min
Rebuild and audit structured-judge evidence
Observe: Case coverage, immutable source hashes, judge dimensions, and veto records
Demo budget: 2 minutes · one contiguous terminal block
---
class: course-terminal
---
Live demo
# Switching to the terminal
~~~bash
$ cd chapter6/user-memory-system-evaluation && python build_63_evidence.py
~~~
Run the command(s), narrate decisions, and point to the observation—not just the output.
---
# What the evidence supports
Finding 1
Paired case-level analysis is more sensitive than comparing unrelated averages.
Finding 2
Traces make a regression actionable by revealing the failing mechanism.
Finding 3
Evaluation becomes production infrastructure when it controls release and rollback.
---
layout: center
---
Where the claim stops
# Boundary condition
Statistical significance does not imply practical importance, dataset validity, or simulation fidelity.
---
layout: center
---
Engineering takeaway
# Design rule
Ship only deltas that are repeatable, practically meaningful, slice-safe, and traceable to a plausible mechanism.
---
# Continue the experiment
---
layout: center
class: text-center
---
Pause and apply
# Your turn
What would make a statistically significant improvement too small or too risky to ship?
---
layout: center
class: text-center
---
Chapter 6 complete · Next · Lesson 26
Use the evaluation environment as the practice ground for changing model behavior.
→
