ai-agent-book/slides/lesson-12.md

theme, title, info, author, transition, mdc, lineNumbers, monaco, aspectRatio, canvasWidth, layout, class

theme	title	info	author	transition	mdc	lineNumbers	monaco	aspectRatio	canvasWidth	layout	class
seriph	Lesson 12 — Why Is One Retrieval Index Never Enough?	English video course for AI Agents in Depth	Bojie Li	slide-left	true	false	false	16/9	980	cover	cover

Build · Chapter 3 · Memory and Knowledge

Why Is One Retrieval Index Never Enough?

Hybrid search, reranking, multimodality, and structured knowledge

Lesson 12 of 42 · 18 minutes · Hybrid Retrieval; Multimodal Extraction; Structured Indexing; Filesystem Paradigm

---

Why this problem matters

Candidate fusion

Merge dense and sparse result sets.

Reranking

Use a stronger model only on a small candidate pool.

Knowledge shape

Trees, graphs, files, images, and tables preserve different structure.

---

Three ideas to keep in view

Hybrid retrieval

Broad recall from multiple retrievers

Neural reranker

More precise ordering at higher per-item cost

Structured index

Represent hierarchy or relationships explicitly

---

The book's visual model

Hybrid retrieval and reranking pipeline

---

Flat chunks vs. Structured knowledge

Flat chunks

• Simple ingestion
• Local passage questions
• Weak global structure

Structured knowledge

• Hierarchies and graphs
• Multi-hop questions
• More governance cost

Choose an index for the questions—not for fashion.

---

Fuse ranks before reranking

dense = dense_index.search(query, k=20)
sparse = bm25.search(query, k=20)
pool = reciprocal_rank_fusion(dense, sparse)
answer_context = reranker.top(query, pool, k=5)

---

Test the claim

3-63 min

Expose every retrieval stage

Observe: Dense candidates, sparse candidates, fusion, reranking, and final rank

3-82 min

Compare RAPTOR and GraphRAG

Observe: Questions favored by hierarchical summaries versus relationship graphs

Demo budget: 5 minutes · one contiguous terminal block

---

class: course-terminal

Live demo

Switching to the terminal

$ uv run python chapter3/retrieval-pipeline/evaluate.py --query "XR-7003"

$ uv run python chapter3/structured-index/main.py demo

Run the command(s), narrate decisions, and point to the observation—not just the output.

---

What the evidence supports

Finding 1

Hybrid retrieval improves recall because its component failures differ.

Finding 2

Reranking spends expensive reasoning on a small, diverse pool.

Finding 3

Structured indexes help only when queries need their encoded structure.

---

Boundary → design rule

More stages increase latency, operational cost, and the number of components that can drift.

Add a retrieval stage only when an evaluation identifies the failure it corrects.

---

Continue the experiment

Experiment 4-2: multimodal strategies chapter4/multimodal-agent/ RAPTOR tree book-en/images/fig3-10.svg GraphRAG graph book-en/images/fig3-11.svg Knowledge-base governance book-en/chapter3.md

---

layout: center class: text-center

Pause and apply

Your turn

Which query type would reveal that your flat index has lost document structure?

---

layout: center class: text-center

Next · Lesson 13

Let the Agent decide whether another retrieval step is necessary.

→