ai-agent-book/slides/lesson-06.md

theme, title, info, author, transition, mdc, lineNumbers, monaco, aspectRatio, canvasWidth, layout, class

theme	title	info	author	transition	mdc	lineNumbers	monaco	aspectRatio	canvasWidth	layout	class
seriph	Lesson 06 — Why Can One Timestamp Make an Agent Slow?	English video course for AI Agents in Depth	Bojie Li	slide-left	true	false	false	16/9	980	cover	cover

Build · Chapter 2 · Context Engineering

Why Can One Timestamp Make an Agent Slow?

Chat templates, attention, KV Cache, and stable prefixes

Lesson 06 of 42 · 19 minutes · KV Cache-Friendly Context Design; Chat Template; Prompt Cache

---

layout: center class: text-center

The central question

Why can a harmless dynamic line near the top of the prompt invalidate most cached computation?

---

Why this problem matters

Token stream

Message objects become one ordered sequence.

Prefix reuse

Matching early tokens reuse previous attention work.

Architecture

Dynamic content placement becomes a systems decision.

---

Three ideas to keep in view

KV Cache

Reuses keys and values within inference

Prompt Cache

Reuses a stable prefix across API requests

Stable prefix

Instructions and tools that do not change turn to turn

---

The book's visual model

KV Cache prefix reuse

---

Cache-friendly vs. Cache-breaking

Cache-friendly

• Stable system prompt
• Stable tool order
• Dynamic state appended late

Cache-breaking

• Timestamp near the front
• Randomized tool order
• Reformatted history

One early mismatch invalidates everything that follows.

---

Move changing state after the stable prefix

static = [system_prompt, stable_tool_schemas]
trajectory = load_messages(session_id)
status = make_dynamic_status(now, progress)
messages = static + trajectory + [status]

---

Test the claim

2-32 min

Compare context-management cache reports

Observe: Prefix hits, recomputation, repeated work, and estimated cost

2-22 min

Generate a small attention view

Observe: A token's weighted access to earlier tokens

Demo budget: 4 minutes · one contiguous terminal block

---

class: course-terminal

Live demo

Switching to the terminal

$ uv run python chapter2/kv-cache/main.py --report

$ uv run python chapter2/attention_visualization/attention_cli.py --prompt "Explain attention in one sentence." --output attention.png

Run the command(s), narrate decisions, and point to the observation—not just the output.

---

What the evidence supports

Finding 1

The API's message abstraction hides an ordered token prefix.

Finding 2

Cache efficiency depends on exact prefix stability.

Finding 3

Correct context management can improve quality and latency together.

---

layout: center

Where the claim stops

Boundary condition

Cache-friendly does not mean never editing context; it means making edits deliberate and localized.

---

layout: center

Engineering takeaway

Design rule

Place stable, frequently reused information first and dynamic information as late as its semantics allow.

---

Continue the experiment

Attention heatmap book-en/images/fig2-7.png Chat-template token structure book-en/images/fig2-8.svg Editable and composable notes book-en/chapter2.md

---

layout: center class: text-center

Pause and apply

Your turn

Which dynamic values in your system prompt silently destroy prefix reuse?

---

layout: center class: text-center

Next · Lesson 07

Even a perfectly cached prompt can fail if its instructions are poorly organized.

→