ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter4/multimodal-agent/.gitignore

@@ -0,0 +1,7 @@
+# 由 create_sample.py 离线生成的样例文件
+test_files/
+
+# Python 缓存与环境
+__pycache__/
+*.pyc
+.env

chapter4/multimodal-agent/README.md

@@ -0,0 +1,307 @@
+# Multimodal Agent — Three Extraction Paradigms / 多模态 Agent——三种抽取范式对比
+
+> Companion material for *AI Agents in Depth*, Chapter 4 — **Experiment 4-2**: native multimodal vs extract-to-text vs tool-based analysis.  
+> 配套《深入理解 AI Agent》第 4 章 **实验 4-2**：原生多模态 vs 先抽文本 vs 工具化分析。
+
+← [Chapter 4 index / 返回第 4 章目录](../README.md)
+
+---
+
+## English
+
+### Features — three extraction modes
+
+1. **Native Multimodality**: model built-in multimodal  
+   - Gemini 2.5 Pro: PDF, image, audio  
+   - GPT-5/GPT-4o: images (OpenAI multimodal format)  
+   - Doubao 1.6: images  
+
+2. **Extract to Text**: convert first, then reason  
+   - PDF OCR (Gemini or GPT-5)  
+   - Image captions (GPT-5 or Doubao 1.6)  
+   - Audio: Whisper or Gemini  
+
+3. **Multimodal analysis tools**: add-on for follow-ups  
+   - Image / audio / PDF analysis tools  
+
+### Architecture
+
+```
+MultimodalAgent
+├── Configuration (config.py)
+├── Agent Core (agent.py) — messages, history, modes, streaming
+└── Multimodal Tools — image, audio, PDF analysis
+```
+
+### Installation
+
+```bash
+# From the repository root: use the shared Chapter 4 environment
+uv sync --locked --python 3.12 --extra ch3
+
+# Activate it before changing directories:
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+# pip fallback when uv is not installed:
+# python -m pip install -e ".[ch3]"
+
+cd chapter4/multimodal-agent
+
+# Exact legacy parity path, including python-magic file sniffing:
+# python -m pip install -r requirements.txt
+
+cp env.example .env
+# Edit .env with API keys
+export $(cat .env | xargs)   # optional on Unix
+```
+
+### Quick offline start (no API key)
+
+Generate a chart-bearing sample so Experiment 4-2 is measurable—**exact quarterly figures live only in the chart bars**, not surrounding text:
+
+```bash
+python create_sample.py           # or: python demo.py --generate-sample
+# → test_files/sample_chart.png, test_files/sample_report.pdf
+```
+
+Then compare three paradigms (needs vision API key):
+
+```bash
+python demo.py \
+  --file test_files/sample_chart.png \
+  --query "Which quarter had the highest revenue, and what was the exact value?" \
+  --model gpt-5.6-luna
+```
+
+Chinese `--help` on `demo.py` / `main.py` / `create_sample.py`.
+
+### Usage
+
+#### Interactive
+
+```bash
+python main.py --interactive
+```
+
+Commands: `/file <path>`, `/mode <native|extract_to_text>`, `/model <name>`, `/tools <on|off>`, `/history`, `/clear`, `/quit`.
+
+#### Single file
+
+```bash
+python main.py --file document.pdf --query "What is the main topic?"
+python main.py --mode extract_to_text --file image.jpg --query "Describe this image"
+python main.py --tools --mode extract_to_text --file audio.mp3 --query "What's the content?"
+```
+
+#### Programmatic
+
+```python
+import asyncio
+from agent import MultimodalAgent, MultimodalContent
+from config import ExtractionMode
+
+async def example():
+    agent = MultimodalAgent(
+        model="gemini-3.5-flash",
+        mode=ExtractionMode.NATIVE,
+        enable_tools=True
+    )
+    content = MultimodalContent(type="pdf", path="document.pdf")
+    result = await agent.process_multimodal_content(content, "Summarize this document")
+    print(result)
+    async for chunk in agent.chat("Tell me more about the key points", stream=True):
+        print(chunk, end="", flush=True)
+
+asyncio.run(example())
+```
+
+### Demo comparison
+
+```bash
+python demo.py --file document.pdf --query "What are the key findings?" --model gpt-5.6-luna
+python demo.py document.pdf "What are the key findings?"   # positional still works
+python demo.py --file test_files/sample_chart.png \
+  --query "Which quarter had the highest revenue?" \
+  --model gpt-5.6-luna --skip-model-comparison --output result.txt
+```
+
+Runs: (1) native (2) extract-to-text (3) extract + tools (4) cross-model unless skipped.
+
+| Flag | Description |
+|------|-------------|
+| `--file` / positional | Image / PDF / audio |
+| `--query` / positional | Question |
+| `--model` | Default `gemini-3.5-flash` |
+| `--skip-model-comparison` | Only three-paradigm compare |
+| `--generate-sample` | Offline sample then exit |
+| `--output`, `-o` | Transcript file |
+
+### Mode comparison
+
+| Mode | Advantages | Disadvantages | Best for |
+|------|------------|---------------|----------|
+| **Native** | Full context; better vision | Limited models; more tokens | Mixed complex docs |
+| **Extract to Text** | Any text model; cacheable | Loses visual context | Text-heavy / cost |
+| **With Tools** | Follow-ups; selective depth | More API calls | Interactive Q&A |
+
+### Supported files / models
+
+- PDF (best native Gemini), images (JPEG/PNG/GIF/BMP/WebP), audio (MP3/WAV/M4A/FLAC/AAC/OGG)  
+- Size limits: PDF/images 20MB, audio 25MB  
+
+| Model | Native PDF | Native Image | Native Audio | Extract | Tools |
+|-------|------------|--------------|--------------|---------|-------|
+| Gemini 2.5 Pro | ✅ | ✅ | ✅ | ✅ | ✅ |
+| GPT-5/GPT-4o | ❌ | ✅ | ❌ | ✅ | ✅ |
+| Doubao 1.6 | ❌ | ✅ | ❌ | ✅ | ✅ |
+
+### API keys
+
+- `GOOGLE_API_KEY` or `GEMINI_API_KEY` — PDF/audio native  
+- `OPENAI_API_KEY` — GPT + Whisper  
+- `DOUBAO_API_KEY` or `ARK_API_KEY`  
+
+### Testing / best practices
+
+```bash
+python test_multimodal.py
+```
+
+Prefer native when vision/audio fidelity matters; extract-to-text for cost/cache; tools for multi-turn. Validate files and keys; handle rate limits.
+
+### License
+
+MIT License — educational project.
+
+---
+
+## 中文
+
+### 功能——三种抽取模式
+
+1. **原生多模态**：直接用模型内置能力（Gemini PDF/图/音频；GPT/豆包图像等）  
+2. **先抽文本再推理**：PDF OCR、图像描述、Whisper/Gemini 转写  
+3. **多模态分析工具**：跟进问题的图像 / 音频 / PDF 工具  
+
+### 架构
+
+```
+MultimodalAgent
+├── Configuration (config.py)
+├── Agent Core (agent.py)
+└── Multimodal Tools
+```
+
+### 安装
+
+```bash
+# 在仓库根目录使用统一的第 4 章环境
+uv sync --locked --python 3.12 --extra ch3
+
+# 切换目录前先激活环境：
+# macOS/Linux：
+source .venv/bin/activate
+# Windows PowerShell：.venv\Scripts\Activate.ps1
+# Windows cmd：.venv\Scripts\activate.bat
+
+# 未安装 uv 时可用 pip 兜底：
+# python -m pip install -e ".[ch3]"
+
+cd chapter4/multimodal-agent
+
+# 精确复现旧版单项目环境，含 python-magic 文件类型检测：
+# python -m pip install -r requirements.txt
+
+cp env.example .env
+# 编辑 API Key
+export $(cat .env | xargs)   # Unix 可选
+```
+
+### 离线快速开始（无需 API Key）
+
+生成带图表的样例报告——**精确季度数字只在柱状图里**，方便测三种范式取舍：
+
+```bash
+python create_sample.py
+```
+
+再对比三种范式（需视觉 API Key）：
+
+```bash
+python demo.py \
+  --file test_files/sample_chart.png \
+  --query "Which quarter had the highest revenue, and what was the exact value?" \
+  --model gpt-5.6-luna
+```
+
+各 CLI 均有中文 `--help`。
+
+### 用法
+
+```bash
+python main.py --interactive
+# /file /mode /model /tools /history /clear /quit
+
+python main.py --file document.pdf --query "What is the main topic?"
+python main.py --mode extract_to_text --file image.jpg --query "Describe this image"
+python main.py --tools --mode extract_to_text --file audio.mp3 --query "What's the content?"
+```
+
+程序化用法见 English 节 `asyncio` 示例。
+
+### 对比演示
+
+```bash
+python demo.py --file document.pdf --query "What are the key findings?" --model gpt-5.6-luna
+python demo.py --file test_files/sample_chart.png \
+  --query "Which quarter had the highest revenue?" \
+  --model gpt-5.6-luna --skip-model-comparison --output result.txt
+```
+
+| 标志 | 说明 |
+|------|------|
+| `--file` | 多模态文件 |
+| `--query` | 问题 |
+| `--model` | 默认 `gemini-3.5-flash` |
+| `--skip-model-comparison` | 只做三范式对比 |
+| `--generate-sample` | 离线生成样例后退出 |
+| `--output`, `-o` | 保存完整记录 |
+
+### 模式对比
+
+| 模式 | 优势 | 劣势 | 适用 |
+|------|------|------|------|
+| **原生** | 上下文与视觉完整 | 模型支持有限、token 多 | 复杂混排文档 |
+| **抽文本** | 通用、可缓存 | 丢视觉细节 | 文本向 / 控成本 |
+| **工具** | 可追问、按需深挖 | 多次 API | 交互式问答 |
+
+### 文件与模型能力
+
+支持 PDF / 常见图像 / 常见音频；大小限制 PDF/图 20MB、音频 25MB。能力矩阵与 English 表相同。
+
+### API Key
+
+- `GOOGLE_API_KEY` 或 `GEMINI_API_KEY`  
+- `OPENAI_API_KEY`  
+- `DOUBAO_API_KEY` 或 `ARK_API_KEY`  
+
+### 测试
+
+```bash
+python test_multimodal.py
+```
+
+### 许可
+
+MIT — 教学项目。
+
+---
+
+## Notes / 说明
+
+### OpenRouter 通用回退 / Universal OpenRouter fallback
+
+Chat / vision can route via OpenRouter when `OPENROUTER_API_KEY` is set and primary keys are missing. **Audio transcription (Whisper) and native-PDF extraction still need direct OpenAI/Gemini keys.**

chapter4/multimodal-agent/campaign.py

@@ -0,0 +1,401 @@
+#!/usr/bin/env python3
+"""Live three-paradigm comparison for Chapter 4 Experiment 4-2.
+
+The PNG chart and the PDF page containing that chart are each submitted to the
+same two questions through native vision, local text extraction followed by a
+text-only model, and an agent that decides whether to invoke a vision tool.
+Every provider call is checkpointed immediately and later copied into the
+immutable campaign receipts.
+"""
+
+from __future__ import annotations
+
+import argparse
+import base64
+import json
+import os
+import re
+import shutil
+import subprocess
+import sys
+import tempfile
+import time
+from pathlib import Path
+from typing import Any
+
+from openai import OpenAI
+
+PROJECT_DIR = Path(__file__).resolve().parent
+CHAPTER_DIR = PROJECT_DIR.parent
+sys.path.insert(0, str(CHAPTER_DIR))
+
+from experiment_utils import ChatRecorder, jsonable, sha256_file, write_campaign_evidence  # noqa: E402
+
+ARK_ENDPOINT = "https://ark.cn-beijing.volces.com/api/v3"
+MOONSHOT_ENDPOINT = "https://api.moonshot.cn/v1"
+SEED = 37
+QUESTIONS = [
+    {
+        "id": "highest",
+        "question": "Which quarter had the highest revenue, and what was the exact value?",
+        "expected": "Q4, $180M",
+        "required_patterns": [r"\bQ4\b", r"(?:\$\s*)?180\s*M"],
+    },
+    {
+        "id": "lowest_gap",
+        "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+        "expected": "Q3, $95M; Q4 exceeded it by $85M",
+        "required_patterns": [r"\bQ3\b", r"(?:\$\s*)?95\s*M", r"(?:\$\s*)?85\s*M"],
+    },
+]
+TOOL = {
+    "type": "function",
+    "function": {
+        "name": "inspect_visual",
+        "description": "Inspect the original chart or PDF page when exact visual, spatial, or numeric evidence is needed.",
+        "parameters": {
+            "type": "object",
+            "properties": {"question": {"type": "string"}},
+            "required": ["question"],
+        },
+    },
+}
+
+
+class CheckpointRecorder(ChatRecorder):
+    def __init__(self, *args: Any, checkpoint: Path, **kwargs: Any):
+        super().__init__(*args, **kwargs)
+        self.checkpoint = checkpoint
+        self.checkpoint.parent.mkdir(parents=True, exist_ok=True)
+
+    def create(self, *, purpose: str, **request: Any) -> Any:
+        try:
+            return super().create(purpose=purpose, **request)
+        finally:
+            self.checkpoint.write_text(
+                json.dumps(self.calls, ensure_ascii=False, indent=2) + "\n", encoding="utf-8"
+            )
+
+
+def data_url(path: Path) -> str:
+    mime = "image/png" if path.suffix.lower() == ".png" else "image/jpeg"
+    return f"data:{mime};base64,{base64.b64encode(path.read_bytes()).decode('ascii')}"
+
+
+def local_extract(kind: str, original: Path) -> tuple[str, dict[str, Any]]:
+    started = time.perf_counter()
+    if kind == "png":
+        command = ["tesseract", str(original), "stdout", "--psm", "6"]
+    else:
+        command = ["pdftotext", "-layout", str(original), "-"]
+    proc = subprocess.run(command, text=True, capture_output=True, check=True)
+    return proc.stdout.strip(), {
+        "command": command,
+        "stderr": proc.stderr,
+        "latency_ms": round((time.perf_counter() - started) * 1000, 3),
+    }
+
+
+def render_pdf(pdf: Path, output: Path) -> None:
+    prefix = output.with_suffix("")
+    subprocess.run(
+        ["pdftoppm", "-png", "-singlefile", "-r", "180", str(pdf), str(prefix)],
+        check=True,
+        capture_output=True,
+        text=True,
+    )
+
+
+def answer_text(recorder: CheckpointRecorder, model: str, context: str, question: str, purpose: str) -> str:
+    response = recorder.create(
+        purpose=purpose,
+        model=model,
+        seed=SEED,
+        temperature=0,
+        messages=[
+            {"role": "system", "content": "Answer only from the extracted text. If it lacks the exact visual evidence, say that it is unavailable."},
+            {"role": "user", "content": f"Extracted text:\n{context}\n\nQuestion: {question}"},
+        ],
+    )
+    return response.choices[0].message.content or ""
+
+
+def answer_vision(recorder: CheckpointRecorder, model: str, image: Path, question: str, purpose: str) -> str:
+    response = recorder.create(
+        purpose=purpose,
+        model=model,
+        seed=SEED,
+        temperature=0,
+        messages=[
+            {
+                "role": "user",
+                "content": [
+                    {"type": "text", "text": f"Read the chart carefully. {question} Give exact values and concise supporting visual evidence."},
+                    {"type": "image_url", "image_url": {"url": data_url(image)}},
+                ],
+            }
+        ],
+    )
+    return response.choices[0].message.content or ""
+
+
+def answer_with_tool(
+    recorder: CheckpointRecorder,
+    model: str,
+    extracted: str,
+    image: Path,
+    question: str,
+    artifact_id: str,
+) -> tuple[str, dict[str, Any]]:
+    messages: list[dict[str, Any]] = [
+        {
+            "role": "system",
+            "content": (
+                "You are given a cheap text extraction and one visual-inspection tool. "
+                "Call inspect_visual whenever exact chart values or spatial associations are not explicitly established by the text."
+            ),
+        },
+        {"role": "user", "content": f"Extracted text:\n{extracted}\n\nQuestion: {question}"},
+    ]
+    decision = recorder.create(
+        purpose=f"tool-decision:{artifact_id}",
+        model=model,
+        seed=SEED,
+        temperature=0,
+        messages=messages,
+        tools=[TOOL],
+        tool_choice="auto",
+    )
+    message = decision.choices[0].message
+    calls = list(message.tool_calls or [])
+    trace: dict[str, Any] = {"tool_selected": bool(calls), "decision": jsonable(message), "executions": []}
+    if not calls:
+        return message.content or "", trace
+
+    messages.append(message.model_dump(exclude_none=True))
+    for call in calls:
+        arguments = json.loads(call.function.arguments or "{}")
+        tool_question = arguments.get("question") or question
+        result = answer_vision(
+            recorder,
+            model,
+            image,
+            tool_question,
+            f"tool-vision:{artifact_id}:{call.id}",
+        )
+        trace["executions"].append(
+            {"tool_call_id": call.id, "name": call.function.name, "arguments": arguments, "result": result}
+        )
+        messages.append({"role": "tool", "tool_call_id": call.id, "content": result})
+    final = recorder.create(
+        purpose=f"tool-final:{artifact_id}",
+        model=model,
+        seed=SEED,
+        temperature=0,
+        messages=messages,
+        tools=[TOOL],
+        tool_choice="none",
+    )
+    return final.choices[0].message.content or "", trace
+
+
+def exact_correct(answer: str, patterns: list[str]) -> bool:
+    return all(re.search(pattern, answer, flags=re.IGNORECASE) for pattern in patterns)
+
+
+def judge_answers(recorder: CheckpointRecorder, model: str, rows: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    payload = [
+        {"id": row["id"], "question": row["question"], "reference": row["expected"], "answer": row["answer"]}
+        for row in rows
+    ]
+    response = recorder.create(
+        purpose="external-answer-judge",
+        model=model,
+        seed=SEED,
+        temperature=0,
+        response_format={"type": "json_object"},
+        messages=[
+            {
+                "role": "system",
+                "content": (
+                    "Independently judge chart QA answers. Return JSON {items:[{id,correct,score,reason}]}. "
+                    "Score 1 only if every requested quarter/value/difference matches the reference; otherwise 0."
+                ),
+            },
+            {"role": "user", "content": json.dumps(payload, ensure_ascii=False)},
+        ],
+    )
+    try:
+        return json.loads(response.choices[0].message.content)["items"]
+    except json.JSONDecodeError:
+        return []
+
+
+def tool_version(command: list[str]) -> str:
+    proc = subprocess.run(command, text=True, capture_output=True)
+    return (proc.stdout or proc.stderr).splitlines()[0]
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Experiment 4-2 live multimodal campaign")
+    parser.add_argument("--model", default=os.getenv("ARK_MODEL", "doubao-seed-1-6-250615"))
+    parser.add_argument("--judge-model", default=os.getenv("MULTIMODAL_JUDGE_MODEL", "moonshot-v1-8k"))
+    args = parser.parse_args()
+    ark_key = os.getenv("ARK_API_KEY") or os.getenv("DOUBAO_API_KEY")
+    moonshot_key = os.getenv("MOONSHOT_API_KEY")
+    if not ark_key or not moonshot_key:
+        raise RuntimeError("ARK_API_KEY and MOONSHOT_API_KEY are required")
+
+    checkpoint_dir = PROJECT_DIR / "validation" / "checkpoints"
+    checkpoint_id = time.strftime("%Y%m%dT%H%M%SZ", time.gmtime())
+    ark = CheckpointRecorder(
+        OpenAI(api_key=ark_key, base_url=ARK_ENDPOINT, timeout=120, max_retries=3),
+        "volcengine-ark",
+        ARK_ENDPOINT,
+        checkpoint=checkpoint_dir / f"{checkpoint_id}-ark.json",
+    )
+    judge = CheckpointRecorder(
+        OpenAI(api_key=moonshot_key, base_url=MOONSHOT_ENDPOINT, timeout=120, max_retries=3),
+        "moonshot",
+        MOONSHOT_ENDPOINT,
+        checkpoint=checkpoint_dir / f"{checkpoint_id}-judge.json",
+    )
+
+    chart = PROJECT_DIR / "test_files" / "sample_chart.png"
+    pdf = PROJECT_DIR / "test_files" / "sample_report.pdf"
+    if not chart.exists() or not pdf.exists():
+        subprocess.run([sys.executable, str(PROJECT_DIR / "create_sample.py")], cwd=PROJECT_DIR, check=True)
+
+    with tempfile.TemporaryDirectory() as temp_dir:
+        rendered_pdf = Path(temp_dir) / "sample_report_page.png"
+        render_pdf(pdf, rendered_pdf)
+        artifacts = [
+            ("png", chart, chart),
+            ("pdf", pdf, rendered_pdf),
+        ]
+        rows: list[dict[str, Any]] = []
+        artifact_records = []
+        for kind, original, visual in artifacts:
+            extracted, extraction_receipt = local_extract(kind, original)
+            artifact_records.append(
+                {
+                    "kind": kind,
+                    "source_path": str(original),
+                    "source_sha256": sha256_file(original),
+                    "visual_input": str(visual),
+                    "visual_sha256": sha256_file(visual),
+                    "extracted_text": extracted,
+                    "extraction": extraction_receipt,
+                }
+            )
+            for spec in QUESTIONS:
+                base = {
+                    "artifact": kind,
+                    "question_id": spec["id"],
+                    "question": spec["question"],
+                    "expected": spec["expected"],
+                }
+                started = time.perf_counter()
+                native = answer_vision(ark, args.model, visual, spec["question"], f"native:{kind}:{spec['id']}")
+                rows.append(
+                    {
+                        **base,
+                        "id": f"{kind}:native:{spec['id']}",
+                        "paradigm": "native-multimodal",
+                        "answer": native,
+                        "latency_ms": round((time.perf_counter() - started) * 1000, 3),
+                        "exact_correct": exact_correct(native, spec["required_patterns"]),
+                    }
+                )
+                started = time.perf_counter()
+                text_answer = answer_text(
+                    ark, args.model, extracted, spec["question"], f"extract-text:{kind}:{spec['id']}"
+                )
+                rows.append(
+                    {
+                        **base,
+                        "id": f"{kind}:extract:{spec['id']}",
+                        "paradigm": "extract-to-text",
+                        "answer": text_answer,
+                        "latency_ms": round(extraction_receipt["latency_ms"] + (time.perf_counter() - started) * 1000, 3),
+                        "exact_correct": exact_correct(text_answer, spec["required_patterns"]),
+                    }
+                )
+                started = time.perf_counter()
+                tool_answer, tool_trace = answer_with_tool(
+                    ark, args.model, extracted, visual, spec["question"], f"{kind}:{spec['id']}"
+                )
+                rows.append(
+                    {
+                        **base,
+                        "id": f"{kind}:tool:{spec['id']}",
+                        "paradigm": "tool-on-demand",
+                        "answer": tool_answer,
+                        "latency_ms": round(extraction_receipt["latency_ms"] + (time.perf_counter() - started) * 1000, 3),
+                        "exact_correct": exact_correct(tool_answer, spec["required_patterns"]),
+                        "tool_trace": tool_trace,
+                    }
+                )
+
+        judgements = judge_answers(judge, args.judge_model, rows)
+        judged = {item["id"]: item for item in judgements}
+        for row in rows:
+            row["external_judge"] = judged[row["id"]]
+        summary: dict[str, Any] = {}
+        for paradigm in ("native-multimodal", "extract-to-text", "tool-on-demand"):
+            selected = [row for row in rows if row["paradigm"] == paradigm]
+            summary[paradigm] = {
+                "cases": len(selected),
+                "exact_accuracy": sum(row["exact_correct"] for row in selected) / len(selected),
+                "judge_accuracy": sum(bool(row["external_judge"]["correct"]) for row in selected) / len(selected),
+                "mean_latency_ms": sum(row["latency_ms"] for row in selected) / len(selected),
+            }
+
+        pdf_text = next(item["extracted_text"] for item in artifact_records if item["kind"] == "pdf")
+        tool_rows = [row for row in rows if row["paradigm"] == "tool-on-demand"]
+        acceptance = {
+            "same_two_questions_all_paradigms_and_artifacts": len(rows) == 12,
+            "png_and_pdf_used": {row["artifact"] for row in rows} == {"png", "pdf"},
+            "chart_answers_absent_from_pdf_body_text": not any(
+                value in pdf_text.lower() for value in ("$180", "180m", "$95", "95m", "$85", "85m")
+            ),
+            "real_native_vision_calls": len([call for call in ark.calls if call["purpose"].startswith("native:")]) == 4,
+            "tool_selected_on_demand": all(row["tool_trace"]["tool_selected"] for row in tool_rows),
+            "real_tool_vision_calls": len([call for call in ark.calls if call["purpose"].startswith("tool-vision:")]) >= 4,
+            "external_moonshot_judge": len(judge.calls) == 1 and len(judgements) == len(rows),
+            "all_calls_checkpointed": (checkpoint_dir / f"{checkpoint_id}-ark.json").exists()
+            and (checkpoint_dir / f"{checkpoint_id}-judge.json").exists(),
+        }
+        evidence = {
+            "status": "passed" if all(acceptance.values()) else "failed",
+            "providers": {
+                "vision_answerer": {"provider": "Volcengine Ark", "endpoint": ARK_ENDPOINT, "model": args.model, "seed": SEED},
+                "judge": {"provider": "Moonshot", "endpoint": MOONSHOT_ENDPOINT, "model": args.judge_model, "seed": SEED},
+            },
+            "local_tools": {
+                "tesseract": tool_version(["tesseract", "--version"]),
+                "pdftotext": tool_version(["pdftotext", "-v"]),
+                "pdftoppm": tool_version(["pdftoppm", "-v"]),
+            },
+            "artifacts": artifact_records,
+            "questions": QUESTIONS,
+            "results": rows,
+            "summary": summary,
+            "acceptance": acceptance,
+            "checkpoint_files": [str(ark.checkpoint), str(judge.checkpoint)],
+        }
+        manifest = write_campaign_evidence(
+            PROJECT_DIR,
+            "4-2",
+            evidence,
+            receipts=ark.calls + judge.calls,
+            input_paths=[__file__, PROJECT_DIR / "create_sample.py", chart, pdf],
+        )
+    print(json.dumps(summary, indent=2))
+    print(json.dumps(acceptance, indent=2))
+    print(f"evidence: {manifest['run_dir']}")
+    return 0 if all(acceptance.values()) else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter4/multimodal-agent/config.py

@@ -0,0 +1,167 @@
+"""
+Configuration for Multimodal Agent
+Supports multiple providers and extraction modes
+"""
+import os
+from typing import Optional, Dict, Any
+from dataclasses import dataclass
+from enum import Enum
+from dotenv import load_dotenv
+
+load_dotenv()
+
+
+def _openrouter_model_id(model) -> str:
+    """Map a provider-native model name to an OpenRouter model id, used by the
+    universal OpenRouter fallback. An explicit OPENROUTER_MODEL env var wins.
+    Vision-capable default (gpt-5.6-luna) so image analysis still works."""
+    override = os.getenv("OPENROUTER_MODEL")
+    if override:
+        return override
+    m = (model or "").strip()
+    if not m:
+        return "openai/gpt-5.6-luna"
+    if "/" in m:
+        return m
+    ml = m.lower()
+    if ml.startswith(("gpt-", "o1", "o3", "o4", "chatgpt")):
+        return "openai/" + m
+    if ml.startswith("claude-"):
+        return "anthropic/claude-opus-4.8"
+    if ml.startswith("gemini"):
+        return "google/" + m  # e.g. gemini-3.5-flash -> google/gemini-3.5-flash
+    # Provider-native ids (doubao-*/qwen/...) -> a widely-available vision model.
+    return "openai/gpt-5.6-luna"
+
+
+class ExtractionMode(Enum):
+    """Modes for multimodal content extraction"""
+    NATIVE = "native"  # Use model's native multimodal capabilities
+    EXTRACT_TO_TEXT = "extract_to_text"  # Convert multimodal to text first
+    
+
+class Provider(Enum):
+    """Supported model providers"""
+    GEMINI = "gemini"
+    OPENAI = "openai"
+    DOUBAO = "doubao"
+    
+
+@dataclass
+class ModelConfig:
+    """Configuration for a specific model"""
+    provider: Provider
+    model_name: str
+    api_key: str
+    base_url: Optional[str] = None
+    supports_native_multimodal: bool = True
+    
+
+class Config:
+    """Main configuration class for multimodal agent"""
+    
+    def __init__(self):
+        # Load API keys from environment.
+        # 兼容常见别名：Gemini 官方 SDK 用 GEMINI_API_KEY，旧文档用 GOOGLE_API_KEY，两者都接受；
+        # 豆包/方舟(Ark)的 Key 环境变量常见为 DOUBAO_API_KEY 或 ARK_API_KEY。
+        self.gemini_api_key = os.getenv("GOOGLE_API_KEY") or os.getenv("GEMINI_API_KEY", "")
+        self.openai_api_key = os.getenv("OPENAI_API_KEY", "")
+        self.doubao_api_key = os.getenv("DOUBAO_API_KEY") or os.getenv("ARK_API_KEY", "")
+
+        # Universal OpenRouter fallback: when a model's own provider key is
+        # missing but OPENROUTER_API_KEY is present, route that model through
+        # OpenRouter's OpenAI-compatible endpoint.
+        self.openrouter_api_key = os.getenv("OPENROUTER_API_KEY", "")
+        self.openrouter_base_url = "https://openrouter.ai/api/v1"
+        
+        # Model configurations
+        self.models = {
+            "gemini-3.5-flash": ModelConfig(
+                provider=Provider.GEMINI,
+                model_name="gemini-3.5-flash",
+                api_key=self.gemini_api_key,
+                supports_native_multimodal=True
+            ),
+            "gpt-5": ModelConfig(
+                provider=Provider.OPENAI,
+                model_name="gpt-5",
+                api_key=self.openai_api_key,
+                supports_native_multimodal=True
+            ),
+            "gpt-5.6-luna": ModelConfig(
+                provider=Provider.OPENAI,
+                model_name="gpt-5.6-luna",
+                api_key=self.openai_api_key,
+                supports_native_multimodal=True
+            ),
+            "doubao-1.6": ModelConfig(
+                provider=Provider.DOUBAO,
+                model_name=os.getenv("ARK_MODEL", "doubao-seed-1-6-250615"),
+                api_key=self.doubao_api_key,
+                base_url="https://ark.cn-beijing.volces.com/api/v3",
+                supports_native_multimodal=True
+            )
+        }
+        
+        # Default settings
+        self.default_model = os.getenv("MULTIMODAL_MODEL", "doubao-1.6")
+        self.default_mode = ExtractionMode.NATIVE
+        self.enable_multimodal_tools = False
+        
+        # File size limits (in MB)
+        self.max_pdf_size_mb = 20
+        self.max_image_size_mb = 20
+        self.max_audio_size_mb = 25
+        
+        # Whisper settings for audio transcription
+        self.whisper_model = "whisper-1"
+        
+        # Temperature settings
+        self.temperature = 0.7
+        self.max_tokens = 4096
+        
+    def get_model_config(self, model_name: str) -> ModelConfig:
+        """Get configuration for a specific model"""
+        if model_name not in self.models:
+            raise ValueError(f"Unknown model: {model_name}")
+        return self.models[model_name]
+        
+    def validate_api_keys(self) -> Dict[str, bool]:
+        """Check which API keys are configured"""
+        return {
+            "gemini": bool(self.gemini_api_key),
+            "openai": bool(self.openai_api_key),
+            "doubao": bool(self.doubao_api_key),
+            "openrouter": bool(self.openrouter_api_key)
+        }
+
+    def has_provider_key(self, provider: 'Provider') -> bool:
+        """Whether the direct API key for a provider is configured."""
+        if provider == Provider.OPENAI:
+            return bool(self.openai_api_key)
+        if provider == Provider.DOUBAO:
+            return bool(self.doubao_api_key)
+        if provider == Provider.GEMINI:
+            return bool(self.gemini_api_key)
+        return False
+
+    def use_openrouter(self, provider: 'Provider') -> bool:
+        """True when a model's own provider key is missing but OpenRouter is
+        available -> the call should be routed through OpenRouter."""
+        return (not self.has_provider_key(provider)) and bool(self.openrouter_api_key)
+
+    def openai_client_args(self, model_config: 'ModelConfig'):
+        """Return (client_kwargs, model_name) for an OpenAI-compatible call,
+        applying the universal OpenRouter fallback when needed."""
+        provider = model_config.provider
+        _m = (model_config.model_name or "").lower()
+        _prefer_or = bool(self.openrouter_api_key) and _m.startswith("gpt-5")  # 直连 gpt-5.6 需组织实名，优先 OpenRouter
+        if _prefer_or or self.use_openrouter(provider):
+            return (
+                {"api_key": self.openrouter_api_key, "base_url": self.openrouter_base_url},
+                _openrouter_model_id(model_config.model_name),
+            )
+        if provider == Provider.DOUBAO:
+            return {"api_key": self.doubao_api_key, "base_url": model_config.base_url}, model_config.model_name
+        # OPENAI (and any GEMINI forced through the OpenAI-compatible path)
+        return {"api_key": self.openai_api_key}, model_config.model_name

chapter4/multimodal-agent/create_sample.py

@@ -0,0 +1,134 @@
+"""
+离线样例生成器 (Offline sample generator)
+
+生成一个"含图表的报告"作为多模态样例，用于实验 4-2 对比三种提取范式。
+产物同时包含：
+  - test_files/sample_chart.png   仅图表（图像模态）
+  - test_files/sample_report.pdf  图表 + 文字说明（文档模态，书中的"含图表的 PDF 报告"）
+
+关键设计：图表里的精确数值（如各季度营收）只出现在柱状图上，正文并未逐一写出。
+这样在实验中：
+  - 原生多模态模式可以直接"看懂"柱子读出数值；
+  - 提取为文本模式若用通用描述器转写图像，往往丢失精确数值与空间关系；
+从而让三种范式的取舍可被直接测量，而不是靠猜。
+
+本脚本完全离线，不需要任何 API Key。
+"""
+
+import argparse
+from pathlib import Path
+
+import matplotlib
+matplotlib.use("Agg")  # 无界面后端，纯离线出图
+import matplotlib.pyplot as plt
+
+
+# 图表数据：只在柱状图上标注，正文不重复这些精确数字
+QUARTERS = ["Q1", "Q2", "Q3", "Q4"]
+REVENUE = [120, 150, 95, 180]  # 单位：百万美元 ($M)
+
+
+def create_chart(output_path: Path) -> Path:
+    """用 matplotlib 生成一张柱状图（图像模态样例）。"""
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+
+    fig, ax = plt.subplots(figsize=(6, 4), dpi=150)
+    bars = ax.bar(QUARTERS, REVENUE, color=["#4C72B0", "#55A868", "#C44E52", "#8172B3"])
+
+    # 把精确数值标注在柱子顶端——这些信息只存在于图像里
+    for bar, value in zip(bars, REVENUE):
+        ax.text(
+            bar.get_x() + bar.get_width() / 2,
+            bar.get_height() + 3,
+            f"${value}M",
+            ha="center",
+            va="bottom",
+            fontsize=11,
+            fontweight="bold",
+        )
+
+    ax.set_title("Acme Corp Quarterly Revenue 2024", fontsize=13, fontweight="bold")
+    ax.set_ylabel("Revenue (in $M)")
+    ax.set_ylim(0, 210)
+    ax.grid(axis="y", linestyle="--", alpha=0.4)
+    fig.tight_layout()
+
+    fig.savefig(output_path)
+    plt.close(fig)
+    return output_path
+
+
+def create_report_pdf(chart_path: Path, output_path: Path) -> Path:
+    """把图表和一段文字说明组合成一份 PDF 报告（文档模态样例）。"""
+    try:
+        from reportlab.lib.pagesizes import A4
+        from reportlab.lib.units import cm
+        from reportlab.lib.styles import getSampleStyleSheet
+        from reportlab.platypus import (
+            SimpleDocTemplate,
+            Paragraph,
+            Spacer,
+            Image as RLImage,
+        )
+    except ImportError:
+        print("提示：未安装 reportlab，跳过 PDF 生成（pip install reportlab）。")
+        return None
+
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    styles = getSampleStyleSheet()
+
+    # 正文刻意只给出定性描述，不逐一写出各季度精确数值——数值只在图里
+    body_text = (
+        "This internal report summarizes Acme Corp's revenue performance in 2024. "
+        "Overall the year showed healthy growth, with a mid-year dip followed by a "
+        "strong recovery in the final quarter. The chart below breaks down revenue "
+        "by quarter; management attributes the fourth-quarter surge to the launch of "
+        "the new enterprise product line."
+    )
+
+    doc = SimpleDocTemplate(str(output_path), pagesize=A4)
+    story = [
+        Paragraph("Acme Corp 2024 Revenue Report", styles["Title"]),
+        Spacer(1, 0.4 * cm),
+        Paragraph(body_text, styles["BodyText"]),
+        Spacer(1, 0.6 * cm),
+        RLImage(str(chart_path), width=14 * cm, height=9.3 * cm),
+    ]
+    doc.build(story)
+    return output_path
+
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="离线生成含图表的多模态样例（图像 + PDF 报告），供实验 4-2 使用。无需 API Key。"
+    )
+    parser.add_argument(
+        "--output-dir",
+        default="test_files",
+        help="样例输出目录（默认：test_files）",
+    )
+    parser.add_argument(
+        "--no-pdf",
+        action="store_true",
+        help="只生成 PNG 图表，不生成 PDF 报告",
+    )
+    args = parser.parse_args()
+
+    out_dir = Path(args.output_dir)
+    chart_path = create_chart(out_dir / "sample_chart.png")
+    print(f"已生成图表: {chart_path}")
+
+    if not args.no_pdf:
+        pdf_path = create_report_pdf(chart_path, out_dir / "sample_report.pdf")
+        if pdf_path:
+            print(f"已生成报告: {pdf_path}")
+
+    print(
+        "\n提示：图表上的精确季度营收只存在于图像中，正文并未逐一写出。\n"
+        "可用如下问题对比三种范式（原生 / 提取为文本 / 带工具）：\n"
+        '  "Which quarter had the highest revenue, and what was the exact value?"'
+    )
+
+
+if __name__ == "__main__":
+    main()

chapter4/multimodal-agent/demo.py

@@ -0,0 +1,340 @@
+"""
+Demo script showcasing different extraction techniques
+"""
+
+import argparse
+import asyncio
+import sys
+from pathlib import Path
+from typing import List, Optional, Tuple
+
+from agent import MultimodalAgent, MultimodalContent
+from config import ExtractionMode
+
+
+class _Tee:
+    """Duplicate stdout writes to a file so --output can save the transcript."""
+
+    def __init__(self, stream, file_handle):
+        self._stream = stream
+        self._file = file_handle
+
+    def write(self, data):
+        self._stream.write(data)
+        self._file.write(data)
+
+    def flush(self):
+        self._stream.flush()
+        self._file.flush()
+
+
+async def compare_extraction_modes(file_path: str, query: str, model: str = "gemini-3.5-flash"):
+    """Compare different extraction modes for the same content"""
+    
+    print(f"\n{'='*80}")
+    print(f"COMPARING EXTRACTION MODES")
+    print(f"File: {file_path}")
+    print(f"Query: {query}")
+    print(f"{'='*80}\n")
+    
+    # Determine content type
+    path = Path(file_path)
+    suffix = path.suffix.lower()
+    
+    if suffix == '.pdf':
+        content_type = "pdf"
+    elif suffix in ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp']:
+        content_type = "image"
+    elif suffix in ['.mp3', '.wav', '.m4a', '.flac', '.aac', '.ogg']:
+        content_type = "audio"
+    else:
+        print(f"Unsupported file type: {suffix}")
+        return
+    
+    # Test with native mode (Gemini)
+    print("\n" + "-"*60)
+    print(f"1. NATIVE MULTIMODAL MODE ({model})")
+    print("-"*60)
+
+    agent_native = MultimodalAgent(
+        model=model,
+        mode=ExtractionMode.NATIVE,
+        enable_tools=False
+    )
+    
+    content = MultimodalContent(type=content_type, path=file_path)
+    
+    try:
+        result = await agent_native.process_multimodal_content(content, query)
+        print(result)
+    except Exception as e:
+        print(f"Error: {e}")
+    
+    # Test with extract-to-text mode
+    print("\n" + "-"*60)
+    print("2. EXTRACT TO TEXT MODE")
+    print("-"*60)
+    
+    agent_extract = MultimodalAgent(
+        model=model,
+        mode=ExtractionMode.EXTRACT_TO_TEXT,
+        enable_tools=False
+    )
+    
+    try:
+        # First extract the content
+        print("Extracting content to text...")
+        extracted = await agent_extract._extract_single_content(content)
+        print("\nExtracted text:")
+        print(extracted)
+        
+        # Then answer the query
+        print(f"\nAnswering query with extracted text...")
+        result = await agent_extract._answer_with_context(extracted, query)
+        print(result)
+    except Exception as e:
+        print(f"Error: {e}")
+    
+    # Test with extract-to-text + tools mode
+    print("\n" + "-"*60)
+    print("3. EXTRACT TO TEXT + MULTIMODAL TOOLS")
+    print("-"*60)
+    
+    agent_tools = MultimodalAgent(
+        model=model,
+        mode=ExtractionMode.EXTRACT_TO_TEXT,
+        enable_tools=True
+    )
+    
+    try:
+        print("Using extract-to-text with tools enabled for follow-up questions...")
+        
+        # Initial processing
+        extracted = await agent_tools._extract_single_content(content)
+        print(f"Extracted {len(extracted)} characters")
+        
+        # Simulate a conversation with follow-up
+        async for chunk in agent_tools.chat(query, content, stream=True):
+            print(chunk, end="", flush=True)
+        print()
+        
+        # Follow-up question that might use tools
+        if content_type == "image":
+            follow_up = f"What colors are dominant in the image at {file_path}?"
+        elif content_type == "pdf":
+            follow_up = f"What specific data or figures are mentioned in the PDF at {file_path}?"
+        else:  # audio
+            follow_up = f"What is the tone or mood of the audio at {file_path}?"
+            
+        print(f"\nFollow-up question: {follow_up}")
+        async for chunk in agent_tools.chat(follow_up, None, stream=True):
+            print(chunk, end="", flush=True)
+        print()
+        
+    except Exception as e:
+        print(f"Error: {e}")
+
+
+async def compare_models(file_path: str, query: str):
+    """Compare different models for the same task"""
+    
+    print(f"\n{'='*80}")
+    print(f"COMPARING MODELS")
+    print(f"File: {file_path}")
+    print(f"Query: {query}")
+    print(f"{'='*80}\n")
+    
+    # Determine content type
+    path = Path(file_path)
+    suffix = path.suffix.lower()
+    
+    if suffix == '.pdf':
+        content_type = "pdf"
+    elif suffix in ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp']:
+        content_type = "image"
+    elif suffix in ['.mp3', '.wav', '.m4a', '.flac', '.aac', '.ogg']:
+        content_type = "audio"
+    else:
+        print(f"Unsupported file type: {suffix}")
+        return
+        
+    content = MultimodalContent(type=content_type, path=file_path)
+    
+    # Test with different models
+    models = ["gemini-3.5-flash", "gpt-5.6-luna", "doubao-1.6"]
+    
+    for model in models:
+        print("\n" + "-"*60)
+        print(f"Model: {model}")
+        print("-"*60)
+        
+        try:
+            # Skip if API key not configured
+            from config import Config
+            config = Config()
+            
+            if model == "gemini-3.5-flash" and not config.gemini_api_key:
+                print("Skipping: Gemini API key not configured")
+                continue
+            elif model in ["gpt-5.6-luna", "gpt-5"] and not (config.openai_api_key or config.openrouter_api_key):
+                print("Skipping: OpenAI API key not configured")
+                continue
+            elif model == "doubao-1.6" and not config.doubao_api_key:
+                print("Skipping: Doubao API key not configured")
+                continue
+            
+            agent = MultimodalAgent(
+                model=model,
+                mode=ExtractionMode.NATIVE if content_type != "audio" or model == "gemini-3.5-flash" else ExtractionMode.EXTRACT_TO_TEXT,
+                enable_tools=False
+            )
+            
+            result = await agent.process_multimodal_content(content, query)
+            print(result)
+            
+        except Exception as e:
+            print(f"Error: {e}")
+
+
+async def demo_conversation_with_tools():
+    """Demonstrate a conversation with multimodal tools"""
+    
+    print(f"\n{'='*80}")
+    print("DEMO: CONVERSATION WITH MULTIMODAL TOOLS")
+    print(f"{'='*80}\n")
+    
+    agent = MultimodalAgent(
+        model="gemini-3.5-flash",
+        mode=ExtractionMode.EXTRACT_TO_TEXT,
+        enable_tools=True
+    )
+    
+    # Simulate a conversation
+    conversations = [
+        ("I need help analyzing some documents. I have PDFs, images, and audio files.", None),
+        ("Can you analyze the image at test_files/sample.jpg and tell me what you see?", None),
+        ("Now analyze the PDF at test_files/document.pdf and summarize its main points.", None),
+        ("What's in the audio file at test_files/recording.mp3?", None),
+        ("Based on all these files, what's the common theme?", None)
+    ]
+    
+    for message, content in conversations:
+        print(f"\nUser: {message}")
+        print("Assistant: ", end="", flush=True)
+        
+        try:
+            async for chunk in agent.chat(message, content, stream=True):
+                print(chunk, end="", flush=True)
+            print()
+        except Exception as e:
+            print(f"\nError: {e}")
+            print("(File might not exist - this is a demo)")
+
+
+def build_parser() -> argparse.ArgumentParser:
+    """构建实验 4-2 的命令行接口。"""
+    parser = argparse.ArgumentParser(
+        description=(
+            "实验 4-2：多模态信息提取的三种技术范式对比（原生多模态 / 提取为文本 / 带工具）。\n"
+            "将同一多模态文件和同一问题分别交给三种模式处理，观察表现差异。"
+        ),
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=(
+            "示例：\n"
+            "  # 先离线生成含图表的样例（无需 API Key）\n"
+            "  python demo.py --generate-sample\n"
+            "  # 用生成的图表跑三种范式对比（需要 API Key）\n"
+            "  python demo.py --file test_files/sample_chart.png \\\n"
+            '      --query \"Which quarter had the highest revenue, and what was the exact value?\"\n'
+            "  # 兼容旧写法（位置参数）\n"
+            "  python demo.py document.pdf \"总结这份文档的要点\""
+        ),
+    )
+    parser.add_argument(
+        "file", nargs="?", default=None,
+        help="要处理的多模态文件（图像 / PDF 文档 / 音频）。也可用 --file 指定",
+    )
+    parser.add_argument(
+        "query", nargs="?", default=None,
+        help="向该文件提出的问题。也可用 --query 指定",
+    )
+    parser.add_argument(
+        "--file", dest="file_opt", default=None,
+        help="要处理的多模态文件（等价于位置参数 file）",
+    )
+    parser.add_argument(
+        "--query", dest="query_opt", default=None,
+        help="向该文件提出的问题（等价于位置参数 query）",
+    )
+    parser.add_argument(
+        "--model", default="gemini-3.5-flash",
+        help="原生 / 提取模式使用的模型（默认：gemini-3.5-flash）",
+    )
+    parser.add_argument(
+        "--skip-model-comparison", action="store_true",
+        help="只跑三种范式对比，跳过跨模型对比",
+    )
+    parser.add_argument(
+        "--generate-sample", action="store_true",
+        help="离线生成含图表的样例文件到 test_files/ 后退出（无需 API Key）",
+    )
+    parser.add_argument(
+        "--output", "-o", default=None,
+        help="将完整对比结果同时写入指定文件（如 result.txt）",
+    )
+    return parser
+
+
+async def run_comparison(file_path: str, query: str, model: str, skip_model_comparison: bool):
+    """运行三种范式对比，可选跨模型对比。"""
+    print("="*80)
+    print("MULTIMODAL AGENT DEMO")
+    print("="*80)
+
+    await compare_extraction_modes(file_path, query, model=model)
+    if not skip_model_comparison:
+        await compare_models(file_path, query)
+
+
+async def main():
+    """实验入口：解析参数并运行对比。"""
+    parser = build_parser()
+    args = parser.parse_args()
+
+    # 离线样例生成：不需要 API Key，直接产出图表 + PDF 报告
+    if args.generate_sample:
+        import create_sample
+        sys.argv = ["create_sample.py"]  # 用默认输出目录 test_files/
+        create_sample.main()
+        return
+
+    file_path = args.file_opt or args.file
+    query = args.query_opt or args.query
+
+    # 缺少文件或问题时，回退到无需真实文件的对话演示
+    if not file_path or not query:
+        print("="*80)
+        print("MULTIMODAL AGENT DEMO")
+        print("="*80)
+        print("\n未提供 <file> 与 <query>，改为运行对话演示。")
+        print("用法：python demo.py --file <文件> --query <问题>")
+        print("先生成样例：python demo.py --generate-sample\n")
+        await demo_conversation_with_tools()
+        return
+
+    # 支持 --output：把整段对比结果同时落盘
+    if args.output:
+        with open(args.output, "w", encoding="utf-8") as fh:
+            original_stdout = sys.stdout
+            sys.stdout = _Tee(original_stdout, fh)
+            try:
+                await run_comparison(file_path, query, args.model, args.skip_model_comparison)
+            finally:
+                sys.stdout = original_stdout
+        print(f"\n完整对比结果已写入：{args.output}")
+    else:
+        await run_comparison(file_path, query, args.model, args.skip_model_comparison)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

chapter4/multimodal-agent/env.example

@@ -0,0 +1,24 @@
+# Multimodal Agent Configuration
+# Copy this file to .env and fill in your API keys
+
+# Google Gemini API (GEMINI_API_KEY 亦可，两者都会被读取)
+GOOGLE_API_KEY=your_gemini_api_key_here
+
+# OpenAI API (for GPT-4o / GPT-5 vision and Whisper)
+OPENAI_API_KEY=your_openai_api_key_here
+
+# Doubao API (ByteDance/Volcano Engine; ARK_API_KEY 亦可)
+DOUBAO_API_KEY=your_doubao_api_key_here
+
+# Optional: Mistral API for OCR
+MISTRAL_API_KEY=your_mistral_api_key_here
+
+# OpenRouter universal fallback (optional): if the selected model's own
+# provider key (Gemini/OpenAI/Doubao) is missing but OPENROUTER_API_KEY is set,
+# that model is routed through OpenRouter's OpenAI-compatible endpoint.
+# Model names are mapped automatically (gemini-3.5-flash -> google/gemini-3.5-flash,
+# gpt-5.6-luna -> openai/gpt-5.6-luna; vision-capable default openai/gpt-5.6-luna).
+# Note: audio transcription (Whisper) and native-PDF extraction still need a
+# direct OpenAI/Gemini key. Set OPENROUTER_MODEL to force a specific model id.
+OPENROUTER_API_KEY=your_openrouter_api_key_here
+# OPENROUTER_MODEL=openai/gpt-5.6-luna

chapter4/multimodal-agent/main.py

@@ -0,0 +1,307 @@
+"""
+Main entry point for Multimodal Agent
+Demonstrates different extraction modes and model capabilities
+"""
+
+import asyncio
+import sys
+import argparse
+from pathlib import Path
+from typing import Optional
+
+from agent import MultimodalAgent, MultimodalContent
+from config import ExtractionMode, Config
+
+
+class _Tee:
+    """将 stdout 同时写入终端与文件，用于 --output。"""
+
+    def __init__(self, stream, file_handle):
+        self._stream = stream
+        self._file = file_handle
+
+    def write(self, data):
+        self._stream.write(data)
+        self._file.write(data)
+
+    def flush(self):
+        self._stream.flush()
+        self._file.flush()
+
+
+async def process_file(
+    agent: MultimodalAgent,
+    file_path: str,
+    query: Optional[str] = None
+) -> None:
+    """Process a single file with the agent"""
+    
+    path = Path(file_path)
+    if not path.exists():
+        print(f"Error: File '{file_path}' not found")
+        return
+        
+    # Determine content type
+    suffix = path.suffix.lower()
+    if suffix == '.pdf':
+        content_type = "pdf"
+    elif suffix in ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp']:
+        content_type = "image"
+    elif suffix in ['.mp3', '.wav', '.m4a', '.flac', '.aac', '.ogg']:
+        content_type = "audio"
+    else:
+        print(f"Error: Unsupported file type '{suffix}'")
+        return
+        
+    # Create multimodal content
+    content = MultimodalContent(
+        type=content_type,
+        path=file_path
+    )
+    
+    print(f"\n{'='*60}")
+    print(f"Processing {content_type.upper()}: {path.name}")
+    print(f"Mode: {agent.extraction_mode.value}")
+    print(f"Model: {agent.current_model}")
+    print(f"Multimodal Tools: {'Enabled' if agent.enable_multimodal_tools else 'Disabled'}")
+    print(f"{'='*60}\n")
+    
+    try:
+        # Process content
+        if agent.extraction_mode == ExtractionMode.NATIVE:
+            # Use native multimodal processing
+            result = await agent.process_multimodal_content(content, query)
+            print("Native Processing Result:")
+            print("-" * 40)
+            print(result)
+        else:
+            # Extract to text mode
+            print("Extracting content to text...")
+            extracted = await agent._extract_single_content(content)
+            print("Extracted Text:")
+            print("-" * 40)
+            print(extracted[:1000] + "..." if len(extracted) > 1000 else extracted)
+            
+            if query:
+                print(f"\nAnswering query: {query}")
+                print("-" * 40)
+                answer = await agent._answer_with_context(extracted, query)
+                print(answer)
+                
+    except Exception as e:
+        print(f"Error processing file: {e}")
+
+
+async def interactive_chat(agent: MultimodalAgent) -> None:
+    """Interactive chat session with the agent"""
+    
+    print("\n" + "="*60)
+    print("Interactive Multimodal Chat")
+    print(f"Model: {agent.current_model}")
+    print(f"Mode: {agent.extraction_mode.value}")
+    print(f"Multimodal Tools: {'Enabled' if agent.enable_multimodal_tools else 'Disabled'}")
+    print("="*60)
+    print("\nCommands:")
+    print("  /file <path> - Load a multimodal file")
+    print("  /mode <native|extract_to_text> - Switch extraction mode")
+    print("  /model <model_name> - Switch model")
+    print("  /tools <on|off> - Enable/disable multimodal tools")
+    print("  /history - Show conversation history")
+    print("  /clear - Clear conversation history")
+    print("  /quit - Exit")
+    print("\n")
+    
+    current_content = None
+    
+    while True:
+        try:
+            user_input = input("You: ").strip()
+            
+            if not user_input:
+                continue
+                
+            # Handle commands
+            if user_input.startswith("/"):
+                parts = user_input.split(maxsplit=1)
+                command = parts[0].lower()
+                args = parts[1] if len(parts) > 1 else ""
+                
+                if command == "/quit":
+                    print("Goodbye!")
+                    break
+                    
+                elif command == "/file":
+                    if not args:
+                        print("Usage: /file <path>")
+                        continue
+                        
+                    path = Path(args)
+                    if not path.exists():
+                        print(f"File not found: {args}")
+                        continue
+                        
+                    # Determine content type
+                    suffix = path.suffix.lower()
+                    if suffix == '.pdf':
+                        content_type = "pdf"
+                    elif suffix in ['.jpg', '.jpeg', '.png', '.gif', '.bmp', '.webp']:
+                        content_type = "image"
+                    elif suffix in ['.mp3', '.wav', '.m4a', '.flac', '.aac', '.ogg']:
+                        content_type = "audio"
+                    else:
+                        print(f"Unsupported file type: {suffix}")
+                        continue
+                        
+                    current_content = MultimodalContent(
+                        type=content_type,
+                        path=args
+                    )
+                    
+                    # Extract content immediately if in extract mode
+                    result = await agent.load_and_extract_content(current_content)
+                    print(result)
+                    
+                    # In extract mode, content is already extracted, no need to keep it
+                    if agent.extraction_mode == ExtractionMode.EXTRACT_TO_TEXT:
+                        current_content = None
+                    
+                elif command == "/mode":
+                    if args == "native":
+                        agent.extraction_mode = ExtractionMode.NATIVE
+                        print("Switched to native multimodal mode")
+                    elif args == "extract_to_text":
+                        agent.extraction_mode = ExtractionMode.EXTRACT_TO_TEXT
+                        print("Switched to extract-to-text mode")
+                    else:
+                        print("Usage: /mode <native|extract_to_text>")
+                        
+                elif command == "/model":
+                    if args in agent.config.models:
+                        agent.current_model = args
+                        print(f"Switched to model: {args}")
+                    else:
+                        print(f"Available models: {', '.join(agent.config.models.keys())}")
+                        
+                elif command == "/tools":
+                    if args == "on":
+                        agent.set_multimodal_tools_enabled(True)
+                        print("Multimodal tools enabled")
+                    elif args == "off":
+                        agent.set_multimodal_tools_enabled(False)
+                        print("Multimodal tools disabled")
+                    else:
+                        print("Usage: /tools <on|off>")
+                        
+                elif command == "/history":
+                    history = agent.get_conversation_history()
+                    print("\nConversation History:")
+                    print("-" * 40)
+                    for msg in history:
+                        role = msg["role"].upper()
+                        content = msg["content"]
+                        if isinstance(content, str):
+                            preview = content[:200] + "..." if len(content) > 200 else content
+                            print(f"{role}: {preview}")
+                    print("-" * 40)
+                    
+                elif command == "/clear":
+                    agent.reset_conversation()
+                    current_content = None
+                    print("Conversation history cleared")
+                    
+                else:
+                    print(f"Unknown command: {command}")
+                    
+                continue
+                
+            # Regular chat message
+            print("\nAssistant: ", end="", flush=True)
+            
+            try:
+                async for chunk in agent.chat(user_input, current_content, stream=True):
+                    print(chunk, end="", flush=True)
+                print("\n")
+                
+                # Clear current content after first use
+                current_content = None
+                
+            except Exception as e:
+                print(f"\nError: {e}")
+                
+        except KeyboardInterrupt:
+            print("\n\nInterrupted. Type /quit to exit.")
+            continue
+        except Exception as e:
+            print(f"Error: {e}")
+            continue
+
+
+async def main():
+    """Main entry point"""
+    parser = argparse.ArgumentParser(
+        description="多模态 Agent：对比原生多模态、提取为文本、带工具三种信息提取范式。",
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog=(
+            "示例：\n"
+            "  # 处理图像并提问\n"
+            "  python main.py --file test_files/sample_chart.png --query \"图中哪个季度营收最高？\"\n"
+            "  # 处理 PDF 文档（提取为文本模式）\n"
+            "  python main.py --mode extract_to_text --file report.pdf --query \"总结要点\"\n"
+            "  # 进入交互式对话\n"
+            "  python main.py --interactive"
+        ),
+    )
+    parser.add_argument("--mode", choices=["native", "extract_to_text"], default="native",
+                       help="提取模式：native（原生多模态）或 extract_to_text（提取为文本），默认 native")
+    parser.add_argument("--model", default="gemini-3.5-flash",
+                       help="使用的模型（默认：gemini-3.5-flash）")
+    parser.add_argument("--tools", action="store_true",
+                       help="启用多模态分析工具（analyze_image / analyze_audio / analyze_pdf）")
+    parser.add_argument("--file", help="要处理的单个文件（图像 / PDF 文档 / 音频）")
+    parser.add_argument("--query", help="向该文件提出的问题")
+    parser.add_argument("--output", "-o", help="将处理结果同时写入指定文件")
+    parser.add_argument("--interactive", action="store_true",
+                       help="进入交互式对话会话")
+
+    args = parser.parse_args()
+    
+    # Validate API keys
+    config = Config()
+    api_keys = config.validate_api_keys()
+    
+    print("API Key Status:")
+    for provider, has_key in api_keys.items():
+        status = "✓ Configured" if has_key else "✗ Not configured"
+        print(f"  {provider.capitalize()}: {status}")
+    
+    # Create agent
+    mode = ExtractionMode.NATIVE if args.mode == "native" else ExtractionMode.EXTRACT_TO_TEXT
+    agent = MultimodalAgent(
+        model=args.model,
+        mode=mode,
+        enable_tools=args.tools
+    )
+    
+    # Process based on arguments
+    if args.file:
+        if args.output:
+            # 将结果同时写入文件
+            with open(args.output, "w", encoding="utf-8") as fh:
+                original_stdout = sys.stdout
+                sys.stdout = _Tee(original_stdout, fh)
+                try:
+                    await process_file(agent, args.file, args.query)
+                finally:
+                    sys.stdout = original_stdout
+            print(f"\n处理结果已写入：{args.output}")
+        else:
+            await process_file(agent, args.file, args.query)
+    elif args.interactive:
+        await interactive_chat(agent)
+    else:
+        # Default to interactive mode
+        await interactive_chat(agent)
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

chapter4/multimodal-agent/quickstart.py

@@ -0,0 +1,231 @@
+"""
+Quickstart script for testing multimodal agent
+Creates sample files and demonstrates capabilities
+"""
+
+import asyncio
+import base64
+from pathlib import Path
+import os
+
+from agent import MultimodalAgent, MultimodalContent
+from config import ExtractionMode, Config
+
+
+def create_sample_files():
+    """Create sample files for testing"""
+    
+    # Create test_files directory
+    test_dir = Path("test_files")
+    test_dir.mkdir(exist_ok=True)
+    
+    # Create a simple text-based "image" (SVG)
+    svg_content = """<?xml version="1.0" encoding="UTF-8"?>
+    <svg width="200" height="200" xmlns="http://www.w3.org/2000/svg">
+        <rect x="10" y="10" width="180" height="180" fill="lightblue" stroke="black" stroke-width="2"/>
+        <circle cx="100" cy="100" r="50" fill="yellow" stroke="orange" stroke-width="3"/>
+        <text x="100" y="105" text-anchor="middle" font-size="20" fill="black">Hello AI!</text>
+    </svg>
+    """
+    
+    svg_path = test_dir / "sample.svg"
+    svg_path.write_text(svg_content, encoding="utf-8")
+    print(f"Created: {svg_path}")
+    
+    # Create a simple text file that we'll treat as a "document"
+    doc_content = """
+    # Sample Document for Multimodal Agent Testing
+    
+    ## Introduction
+    This is a test document created for demonstrating the multimodal agent's capabilities.
+    The agent can process this document in different modes:
+    
+    1. **Native Mode**: Direct processing using the model's built-in capabilities
+    2. **Extract to Text**: Convert to text first, then analyze
+    3. **With Tools**: Use specialized tools for detailed analysis
+    
+    ## Key Features
+    - Support for multiple file formats (PDF, images, audio)
+    - Multiple AI model providers (Gemini, OpenAI, Doubao)
+    - Streaming responses for better user experience
+    - Tool calling for advanced analysis
+    
+    ## Technical Details
+    The system uses a unified message format compatible with OpenAI's API structure,
+    making it easy to switch between different providers while maintaining consistency.
+    
+    ## Conclusion
+    This multimodal agent demonstrates state-of-the-art AI capabilities for
+    content understanding and analysis across different modalities.
+    """
+    
+    doc_path = test_dir / "sample_document.txt"
+    doc_path.write_text(doc_content, encoding="utf-8")
+    print(f"Created: {doc_path}")
+    
+    return test_dir
+
+
+async def test_basic_functionality():
+    """Test basic agent functionality"""
+    
+    print("\n" + "="*60)
+    print("QUICKSTART: Testing Multimodal Agent")
+    print("="*60)
+    
+    # Check API keys
+    config = Config()
+    api_keys = config.validate_api_keys()
+    
+    print("\n1. API Key Status:")
+    print("-" * 40)
+    for provider, has_key in api_keys.items():
+        status = "✅ Configured" if has_key else "❌ Not configured"
+        print(f"   {provider.capitalize()}: {status}")
+    
+    if not any(api_keys.values()):
+        print("\n⚠️  Warning: No API keys configured!")
+        print("Please copy env.example to .env and add your API keys.")
+        return
+    
+    # Create sample files
+    print("\n2. Creating Sample Files:")
+    print("-" * 40)
+    test_dir = create_sample_files()
+    
+    # Test with available model
+    if api_keys["gemini"]:
+        model = "gemini-3.5-flash"
+        print(f"\n3. Testing with {model}:")
+        print("-" * 40)
+        
+        agent = MultimodalAgent(
+            model=model,
+            mode=ExtractionMode.EXTRACT_TO_TEXT,
+            enable_tools=False
+        )
+        
+        # Process the text document
+        doc_path = test_dir / "sample_document.txt"
+        content = MultimodalContent(
+            type="text",
+            path=str(doc_path),
+            data=doc_path.read_bytes()
+        )
+        
+        print("Processing sample document...")
+        try:
+            # Simulate as if it's a PDF for demonstration
+            content.type = "pdf"
+            result = await agent._extract_pdf_to_text(content)
+            print("Extracted content preview:")
+            print(result[:300] + "..." if len(result) > 300 else result)
+            
+            # Answer a question
+            print("\nAsking a question about the document...")
+            answer = await agent._answer_with_context(
+                result,
+                "What are the three modes mentioned in the document?"
+            )
+            print("Answer:", answer)
+            
+        except Exception as e:
+            print(f"Error: {e}")
+    
+    elif api_keys["openai"]:
+        model = "gpt-5.6-luna"
+        print(f"\n3. Testing with {model}:")
+        print("-" * 40)
+        
+        agent = MultimodalAgent(
+            model=model,
+            mode=ExtractionMode.EXTRACT_TO_TEXT,
+            enable_tools=False
+        )
+        
+        print("Note: OpenAI models work best with images.")
+        print("For document processing, Gemini is recommended.")
+        
+    else:
+        print("\n3. Skipping tests - no API keys configured")
+
+
+async def test_conversation_mode():
+    """Test conversation mode with streaming"""
+    
+    config = Config()
+    if not config.gemini_api_key and not config.openai_api_key:
+        print("\nSkipping conversation test - no API keys configured")
+        return
+        
+    print("\n" + "="*60)
+    print("4. Testing Conversation Mode")
+    print("="*60)
+    
+    # Use available model
+    if config.gemini_api_key:
+        model = "gemini-3.5-flash"
+    else:
+        model = "gpt-5.6-luna"
+        
+    agent = MultimodalAgent(
+        model=model,
+        mode=ExtractionMode.EXTRACT_TO_TEXT,
+        enable_tools=True
+    )
+    
+    print(f"Using model: {model}")
+    print("Tools: Enabled")
+    print("\nStarting conversation...")
+    print("-" * 40)
+    
+    # Simulate a conversation
+    messages = [
+        "Hello! I'm testing the multimodal agent. Can you explain what you can do?",
+        "What types of files can you process?",
+        "How do the different extraction modes work?"
+    ]
+    
+    for message in messages:
+        print(f"\nUser: {message}")
+        print("Assistant: ", end="", flush=True)
+        
+        try:
+            response_text = ""
+            async for chunk in agent.chat(message, stream=True):
+                print(chunk, end="", flush=True)
+                response_text += chunk
+            print()
+            
+            # Small delay for readability
+            await asyncio.sleep(0.5)
+            
+        except Exception as e:
+            print(f"\nError: {e}")
+            break
+
+
+async def main():
+    """Run all quickstart tests"""
+    
+    print("🚀 Multimodal Agent Quickstart")
+    print("=" * 60)
+    
+    # Run basic tests
+    await test_basic_functionality()
+    
+    # Run conversation test
+    await test_conversation_mode()
+    
+    print("\n" + "="*60)
+    print("✅ Quickstart Complete!")
+    print("="*60)
+    print("\nNext steps:")
+    print("1. Add your API keys to .env file")
+    print("2. Try with your own files: python main.py --file <path> --query <question>")
+    print("3. Start interactive mode: python main.py --interactive")
+    print("4. Run comparisons: python demo.py <file> <query>")
+
+
+if __name__ == "__main__":
+    asyncio.run(main())

chapter4/multimodal-agent/requirements.txt

@@ -0,0 +1,23 @@
+# Core dependencies
+google-genai>=0.1.0  # New Google AI SDK for Gemini
+openai>=1.50.0
+httpx>=0.27.0
+python-dotenv>=1.0.0
+
+# For async support (asyncio 与 mimetypes 属于 Python 标准库，无需安装)
+aiofiles>=24.1.0
+
+# For file handling
+python-magic>=0.4.27
+
+# For the offline sample generator (create_sample.py)
+matplotlib>=3.7.0
+reportlab>=4.0.0
+
+# Testing
+pytest>=8.0.0
+pytest-asyncio>=0.24.0
+pytest-mock>=3.14.0
+
+# Optional: For Mistral OCR
+mistralai>=1.2.0

chapter4/multimodal-agent/test_execute_tool_robust.py

@@ -0,0 +1,68 @@
+"""Regression tests: _execute_tool must return an error string to the model
+instead of raising KeyError/JSONDecodeError on malformed LLM tool-call
+arguments (missing required fields or truncated streamed JSON)."""
+import asyncio
+import json
+import os
+import sys
+import types
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+from agent import MultimodalAgent
+
+
+def _make_agent():
+    """Build a MultimodalAgent without __init__ (which needs API keys)."""
+    agent = MultimodalAgent.__new__(MultimodalAgent)
+    calls = []
+
+    async def fake_analyze(path, query):
+        calls.append((path, query))
+        return "analysis ok"
+
+    agent.tools = types.SimpleNamespace(
+        analyze_image=fake_analyze,
+        analyze_audio=fake_analyze,
+        analyze_pdf=fake_analyze,
+    )
+    return agent, calls
+
+
+def _run(agent, name, arguments):
+    tool_call = {"id": "call_1", "function": {"name": name, "arguments": arguments}}
+    return asyncio.run(agent._execute_tool(tool_call))
+
+
+def test_missing_query_returns_error_not_keyerror():
+    agent, calls = _make_agent()
+    result = _run(agent, "analyze_image", json.dumps({"image_path": "cat.png"}))
+    assert result.startswith("Error:")
+    assert calls == []
+
+
+def test_missing_path_returns_error_not_keyerror():
+    agent, calls = _make_agent()
+    result = _run(agent, "analyze_pdf", json.dumps({"query": "what is this?"}))
+    assert result.startswith("Error:")
+    assert calls == []
+
+
+def test_malformed_json_returns_error_not_exception():
+    agent, calls = _make_agent()
+    result = _run(agent, "analyze_audio", '{"audio_path": "a.mp3", "que')
+    assert result.startswith("Error:")
+    assert calls == []
+
+
+def test_valid_arguments_still_call_tool():
+    agent, calls = _make_agent()
+    result = _run(agent, "analyze_image", json.dumps({"image_path": "cat.png", "query": "describe"}))
+    assert result == "analysis ok"
+    assert calls == [("cat.png", "describe")]
+
+
+def test_unknown_tool_still_reported():
+    agent, _ = _make_agent()
+    result = _run(agent, "analyze_video", json.dumps({}))
+    assert "Unknown tool" in result

chapter4/multimodal-agent/test_interactive_tools_toggle.py

@@ -0,0 +1,37 @@
+"""Regression tests for the interactive multimodal tools toggle."""
+
+import asyncio
+import os
+import sys
+from unittest.mock import patch
+
+sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
+
+from agent import MultimodalAgent, MultimodalTools
+from main import interactive_chat
+
+
+def _run_commands(agent, *commands):
+    with patch("builtins.input", side_effect=commands):
+        asyncio.run(interactive_chat(agent))
+
+
+def test_tools_on_configures_complete_tool_state():
+    agent = MultimodalAgent(enable_tools=False)
+
+    _run_commands(agent, "/tools on", "/quit")
+
+    assert agent.enable_multimodal_tools is True
+    assert isinstance(agent.tools, MultimodalTools)
+    assert {
+        definition["function"]["name"] for definition in agent.tool_definitions
+    } == {"analyze_image", "analyze_audio", "analyze_pdf"}
+
+
+def test_tools_off_disables_tool_execution():
+    agent = MultimodalAgent(enable_tools=True)
+
+    _run_commands(agent, "/tools off", "/quit")
+
+    assert agent.enable_multimodal_tools is False
+    assert agent.tools is None

chapter4/multimodal-agent/test_multimodal.py

@@ -0,0 +1,229 @@
+"""
+Test script for multimodal agent functionality
+"""
+
+import asyncio
+import unittest
+from unittest.mock import Mock, patch, AsyncMock
+from pathlib import Path
+
+from agent import MultimodalAgent, MultimodalContent, MultimodalTools, Message
+from config import ExtractionMode, Provider
+
+
+class TestMultimodalContent(unittest.TestCase):
+    """Test MultimodalContent class"""
+    
+    def test_content_creation(self):
+        """Test creating multimodal content"""
+        content = MultimodalContent(
+            type="image",
+            path="test.jpg",
+            mime_type="image/jpeg"
+        )
+        
+        self.assertEqual(content.type, "image")
+        self.assertEqual(content.path, "test.jpg")
+        self.assertEqual(content.mime_type, "image/jpeg")
+        
+    def test_get_base64(self):
+        """Test base64 encoding"""
+        content = MultimodalContent(
+            type="text",
+            data=b"Hello World"
+        )
+        
+        base64_str = content.get_base64()
+        self.assertEqual(base64_str, "SGVsbG8gV29ybGQ=")
+
+
+class TestMessage(unittest.TestCase):
+    """Test Message class"""
+    
+    def test_message_creation(self):
+        """Test creating messages"""
+        msg = Message(
+            role="user",
+            content="Hello"
+        )
+        
+        self.assertEqual(msg.role, "user")
+        self.assertEqual(msg.content, "Hello")
+        
+    def test_message_to_dict(self):
+        """Test converting message to dictionary"""
+        msg = Message(
+            role="assistant",
+            content="Hi there",
+            tool_calls=[{"id": "1", "function": {"name": "test"}}]
+        )
+        
+        msg_dict = msg.to_dict()
+        self.assertEqual(msg_dict["role"], "assistant")
+        self.assertEqual(msg_dict["content"], "Hi there")
+        self.assertIn("tool_calls", msg_dict)
+
+
+class TestMultimodalAgent(unittest.IsolatedAsyncioTestCase):
+    """Test MultimodalAgent class"""
+    
+    async def test_agent_initialization(self):
+        """Test agent initialization"""
+        agent = MultimodalAgent(
+            model="gemini-3.5-flash",
+            mode=ExtractionMode.NATIVE,
+            enable_tools=False
+        )
+        
+        self.assertEqual(agent.current_model, "gemini-3.5-flash")
+        self.assertEqual(agent.extraction_mode, ExtractionMode.NATIVE)
+        self.assertFalse(agent.enable_multimodal_tools)
+        self.assertIsNone(agent.tools)
+        
+    async def test_agent_with_tools(self):
+        """Test agent initialization with tools"""
+        agent = MultimodalAgent(
+            model="gemini-3.5-flash",
+            mode=ExtractionMode.EXTRACT_TO_TEXT,
+            enable_tools=True
+        )
+        
+        self.assertTrue(agent.enable_multimodal_tools)
+        self.assertIsNotNone(agent.tools)
+        self.assertEqual(len(agent.tool_definitions), 3)
+        
+    async def test_conversation_history(self):
+        """Test conversation history management"""
+        agent = MultimodalAgent()
+        
+        # Add messages
+        agent.add_message(Message(role="user", content="Hello"))
+        agent.add_message(Message(role="assistant", content="Hi"))
+        
+        history = agent.get_conversation_history()
+        self.assertEqual(len(history), 2)
+        self.assertEqual(history[0]["role"], "user")
+        self.assertEqual(history[1]["role"], "assistant")
+        
+        # Reset conversation
+        agent.reset_conversation()
+        history = agent.get_conversation_history()
+        self.assertEqual(len(history), 0)
+        
+    @patch('agent.genai.Client')
+    async def test_extract_pdf_to_text(self, mock_client_class):
+        """Test PDF extraction to text"""
+        agent = MultimodalAgent(mode=ExtractionMode.EXTRACT_TO_TEXT)
+        
+        # Mock Gemini response
+        mock_response = Mock()
+        mock_response.candidates = []
+        mock_response.text = "Extracted PDF text"
+        mock_client = Mock()
+        mock_client.models.generate_content.return_value = mock_response
+        mock_client_class.return_value = mock_client
+        
+        content = MultimodalContent(
+            type="pdf",
+            data=b"PDF content"
+        )
+        
+        result = await agent._extract_pdf_to_text(content)
+        self.assertEqual(result, "Extracted PDF text")
+        
+    @patch('agent.AsyncOpenAI')
+    async def test_extract_image_to_text(self, mock_openai_class):
+        """Test image extraction to text"""
+        agent = MultimodalAgent(mode=ExtractionMode.EXTRACT_TO_TEXT)
+        
+        # Mock OpenAI response
+        mock_client = AsyncMock()
+        mock_response = AsyncMock()
+        mock_choice = Mock()
+        mock_message = Mock()
+        mock_message.content = "Image description"
+        mock_choice.message = mock_message
+        mock_response.choices = [mock_choice]
+        mock_client.chat.completions.create.return_value = mock_response
+        mock_openai_class.return_value = mock_client
+        
+        content = MultimodalContent(
+            type="image",
+            data=b"Image data",
+            mime_type="image/jpeg"
+        )
+        
+        result = await agent._extract_image_to_text(content)
+        self.assertEqual(result, "Image description")
+        
+    @patch('agent.genai.Client')
+    async def test_process_native_gemini(self, mock_client_class):
+        """Test native Gemini processing"""
+        agent = MultimodalAgent(
+            model="gemini-3.5-flash",
+            mode=ExtractionMode.NATIVE
+        )
+        
+        # Mock Gemini response
+        mock_response = Mock()
+        mock_response.candidates = []
+        mock_response.text = "Gemini analysis result"
+        mock_client = Mock()
+        mock_client.models.generate_content.return_value = mock_response
+        mock_client_class.return_value = mock_client
+        
+        content = MultimodalContent(
+            type="pdf",
+            data=b"PDF content"
+        )
+        
+        result = await agent._process_native_gemini(content, "Analyze this")
+        self.assertEqual(result, "Gemini analysis result")
+
+
+class TestMultimodalTools(unittest.IsolatedAsyncioTestCase):
+    """Test MultimodalTools class"""
+    
+    async def test_tools_initialization(self):
+        """Test tools initialization"""
+        agent = MultimodalAgent(enable_tools=True)
+        tools = MultimodalTools(agent)
+        
+        self.assertEqual(tools.agent, agent)
+        
+    @patch('agent.AsyncOpenAI')
+    async def test_analyze_image_tool(self, mock_openai_class):
+        """Test image analysis tool"""
+        agent = MultimodalAgent(enable_tools=True)
+        
+        # Mock OpenAI response
+        mock_client = AsyncMock()
+        mock_response = AsyncMock()
+        mock_choice = Mock()
+        mock_message = Mock()
+        mock_message.content = "Image analysis"
+        mock_choice.message = mock_message
+        mock_response.choices = [mock_choice]
+        mock_client.chat.completions.create.return_value = mock_response
+        mock_openai_class.return_value = mock_client
+        
+        # Create temporary test image
+        import tempfile
+        with tempfile.NamedTemporaryFile(suffix='.jpg', delete=False) as tmp:
+            tmp.write(b"test image data")
+            tmp_path = tmp.name
+            
+        try:
+            result = await agent.tools.analyze_image(tmp_path, "What's in this image?")
+            self.assertEqual(result, "Image analysis")
+        finally:
+            Path(tmp_path).unlink()
+
+
+def run_tests():
+    """Run all tests"""
+    unittest.main(argv=[''], exit=False, verbosity=2)
+
+
+if __name__ == "__main__":
+    run_tests()

chapter4/multimodal-agent/validation/checkpoints/20260729T185104Z-judge.json

@@ -0,0 +1,67 @@
+[
+  {
+    "purpose": "external-answer-judge",
+    "provider": "moonshot",
+    "endpoint": "https://api.moonshot.cn/v1",
+    "started_at": "2026-07-29T18:54:20.300357+00:00",
+    "latency_ms": 13009.478,
+    "request": {
+      "model": "moonshot-v1-8k",
+      "seed": 37,
+      "temperature": 0,
+      "response_format": {
+        "type": "json_object"
+      },
+      "messages": [
+        {
+          "role": "system",
+          "content": "Independently judge chart QA answers. Return JSON {items:[{id,correct,score,reason}]}. Score 1 only if every requested quarter/value/difference matches the reference; otherwise 0."
+        },
+        {
+          "role": "user",
+          "content": "[{\"id\": \"png:native:highest\", \"question\": \"Which quarter had the highest revenue, and what was the exact value?\", \"reference\": \"Q4, $180M\", \"answer\": \"The highest revenue was in Q4, with an exact value of $180M. Visual evidence: The Q4 bar is the tallest in the chart, explicitly labeled \\\"$180M\\\" and exceeding the revenue of Q1 ($120M), Q2 ($150M), and Q3 ($95M).\\n\\n**Answer:** Q4 had the highest revenue at $180M.\"}, {\"id\": \"png:extract:highest\", \"question\": \"Which quarter had the highest revenue, and what was the exact value?\", \"reference\": \"Q4, $180M\", \"answer\": \"Q1 had the highest revenue, with an exact value of $180M.\"}, {\"id\": \"png:tool:highest\", \"question\": \"Which quarter had the highest revenue, and what was the exact value?\", \"reference\": \"Q4, $180M\", \"answer\": \"The highest revenue was in Q4 with an exact value of $180M.\"}, {\"id\": \"png:native:lowest_gap\", \"question\": \"Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?\", \"reference\": \"Q3, $95M; Q4 exceeded it by $85M\", \"answer\": \"- **Lowest revenue quarter**: Q3  \\n- **Value**: $95M  \\n- **Q4 exceedance**: Q4 revenue ($180M) minus Q3 revenue ($95M) = $85M  \\n\\n\\n**Visual evidence**: The red bar for Q3 is the shortest, labeled \\\"$95M,\\\" confirming it as the lowest. Q4’s purple bar is labeled \\\"$180M,\\\" the tallest. The vertical difference between Q4 and Q3 bars corresponds to $85M.\"}, {\"id\": \"png:extract:lowest_gap\", \"question\": \"Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?\", \"reference\": \"Q3, $95M; Q4 exceeded it by $85M\", \"answer\": \"The lowest revenue was in Q1 with $95M, and Q4 exceeded it by $85M.\"}, {\"id\": \"png:tool:lowest_gap\", \"question\": \"Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?\", \"reference\": \"Q3, $95M; Q4 exceeded it by $85M\", \"answer\": \"The lowest revenue occurred in Q3, with a value of $95M. Q4's revenue was $180M, exceeding Q3 by $85M ($180M - $95M).  \\n\\n**Answers:**  \\n- Lowest revenue quarter: Q3  \\n- Value: $95M  \\n- Q4 exceeded it by: $85M\"}, {\"id\": \"pdf:native:highest\", \"question\": \"Which quarter had the highest revenue, and what was the exact value?\", \"reference\": \"Q4, $180M\", \"answer\": \"The fourth quarter (Q4) had the highest revenue, with an exact value of $180M.  \\n\\n**Visual evidence**: The Q4 bar in the \\\"Acme Corp Quarterly Revenue 2024\\\" bar graph is the tallest, explicitly labeled \\\"$180M,\\\" exceeding Q1 ($120M), Q2 ($150M), and Q3 ($95M).\"}, {\"id\": \"pdf:extract:highest\", \"question\": \"Which quarter had the highest revenue, and what was the exact value?\", \"reference\": \"Q4, $180M\", \"answer\": \"The fourth quarter had the highest revenue, but the exact value is unavailable.\"}, {\"id\": \"pdf:tool:highest\", \"question\": \"Which quarter had the highest revenue, and what was the exact value?\", \"reference\": \"Q4, $180M\", \"answer\": \"The highest revenue in 2024 was in the fourth quarter (Q4), with an exact value of $180M. This is supported by the visual bar chart labeled \\\"Acme Corp Quarterly Revenue 2024,\\\" which shows Q4 as the tallest bar explicitly marked with \\\"$180M,\\\" exceeding Q1 ($120M), Q2 ($150M), and Q3 ($95M). The report also notes a \\\"strong recovery in the final quarter\\\" attributed to the launch of the new enterprise product line, confirming Q4's surge.\"}, {\"id\": \"pdf:native:lowest_gap\", \"question\": \"Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?\", \"reference\": \"Q3, $95M; Q4 exceeded it by $85M\", \"answer\": \"- **Lowest revenue quarter**: Q3  \\n- **Value**: $95M  \\n- **Q4 exceedance**: Q4 revenue ($180M) minus Q3 revenue ($95M) = $85M  \\n\\n\\n**Visual evidence**: The bar graph shows Q3 with the shortest bar, labeled $95M. Q4’s bar is the tallest, labeled $180M. The vertical gap between Q3 and Q4 bars corresponds to $85M.\"}, {\"id\": \"pdf:extract:lowest_gap\", \"question\": \"Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?\", \"reference\": \"Q3, $95M; Q4 exceeded it by $85M\", \"answer\": \"unavailable\"}, {\"id\": \"pdf:tool:lowest_gap\", \"question\": \"Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?\", \"reference\": \"Q3, $95M; Q4 exceeded it by $85M\", \"answer\": \"The lowest revenue quarter was Q3, with a value of $95M. Q4 revenue was $180M, exceeding Q3 by $85M. \\n\\n**Details**: The visual inspection confirms Q3 (red bar) as the lowest with $95M, while Q4 (purple bar) is the highest at $180M. The difference is calculated as $180M - $95M = $85M.\"}]"
+        }
+      ]
+    },
+    "response": {
+      "id": "chatcmpl-6a6a4c5c0efb1450ac9b2119",
+      "choices": [
+        {
+          "finish_reason": "stop",
+          "index": 0,
+          "logprobs": null,
+          "message": {
+            "content": "{\"items\":[{\"id\": \"png:native:highest\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer.\"}, {\"id\": \"png:extract:highest\", \"correct\": false, \"score\": 0, \"reason\": \"The answer incorrectly states Q1 as the quarter with the highest revenue instead of Q4.\"}, {\"id\": \"png:tool:highest\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer.\"}, {\"id\": \"png:native:lowest_gap\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer.\"}, {\"id\": \"png:extract:lowest_gap\", \"correct\": false, \"score\": 0, \"reason\": \"The answer incorrectly states Q1 as the quarter with the lowest revenue instead of Q3.\"}, {\"id\": \"png:tool:lowest_gap\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer.\"}, {\"id\": \"pdf:native:highest\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer.\"}, {\"id\": \"pdf:extract:highest\", \"correct\": false, \"score\": 0, \"reason\": \"The answer states that the exact value is unavailable, which contradicts the reference answer that provides the exact value of $180M.\"}, {\"id\": \"pdf:tool:highest\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer and providing additional context.\"}, {\"id\": \"pdf:native:lowest_gap\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer.\"}, {\"id\": \"pdf:extract:lowest_gap\", \"correct\": false, \"score\": 0, \"reason\": \"The answer states that the information is unavailable, which contradicts the reference answer that provides the exact values and the difference.\"}, {\"id\": \"pdf:tool:lowest_gap\", \"correct\": true, \"score\": 1, \"reason\": \"The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer.\"}]}",
+            "refusal": null,
+            "role": "assistant",
+            "annotations": null,
+            "audio": null,
+            "function_call": null,
+            "tool_calls": null
+          }
+        }
+      ],
+      "created": 1785351261,
+      "model": "moonshot-v1-8k",
+      "object": "chat.completion",
+      "service_tier": null,
+      "system_fingerprint": null,
+      "usage": {
+        "completion_tokens": 781,
+        "prompt_tokens": 1583,
+        "total_tokens": 2364,
+        "completion_tokens_details": null,
+        "prompt_tokens_details": null
+      }
+    },
+    "usage": {
+      "completion_tokens": 781,
+      "prompt_tokens": 1583,
+      "total_tokens": 2364,
+      "completion_tokens_details": null,
+      "prompt_tokens_details": null
+    },
+    "response_model": "moonshot-v1-8k",
+    "response_id": "chatcmpl-6a6a4c5c0efb1450ac9b2119"
+  }
+]

chapter4/multimodal-agent/validation/latest.json

@@ -0,0 +1,65 @@
+{
+  "schema_version": "chapter3-evidence-v1",
+  "experiment": "4-2",
+  "run_id": "20260729T185433Z-4_2-e028c9db",
+  "created_at": "2026-07-29T18:54:33.367744+00:00",
+  "status": "passed",
+  "run_dir": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/validation/runs/20260729T185433Z-4_2-e028c9db",
+  "artifacts": {
+    "evidence.json": "024358e7b58266eedbedda3cc880a61462c119a89c58e475f4c007dbcb8ff144",
+    "receipts.json": "2aa961a1f7527417dd0d25ac6e542aaad9dc2ffaa7b40a4a450cb5fffe86ceac",
+    "manifest.json": "1a9cc7bfd48717e73a03ebbde7fd786c7da2811a15267715a3794c0f1220362e"
+  },
+  "inputs": [
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/campaign.py",
+      "sha256": "6f8162a0e124cef1acada4fbfadc6a1427aa487be24670aaceba2d668820ee3c",
+      "bytes": 16104
+    },
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/create_sample.py",
+      "sha256": "1f4fcd0e119a76a74c06744a585d31bc4bd4f755f813f8b2518f77127eaa1e72",
+      "bytes": 4846
+    },
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_chart.png",
+      "sha256": "eaf8ac53fbc65a4e6ce0611d441cd12ddd2ed48ef8a727a9d6f22aef70605f39",
+      "bytes": 38417
+    },
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_report.pdf",
+      "sha256": "db4828a20a12ad70dab5c196d23e4fa407f75c4e3cff2bda22c401de65cb9c2e",
+      "bytes": 46368
+    }
+  ],
+  "summary": {
+    "native-multimodal": {
+      "cases": 4,
+      "exact_accuracy": 1.0,
+      "judge_accuracy": 1.0,
+      "mean_latency_ms": 8011.01425
+    },
+    "extract-to-text": {
+      "cases": 4,
+      "exact_accuracy": 0.0,
+      "judge_accuracy": 0.0,
+      "mean_latency_ms": 10867.427
+    },
+    "tool-on-demand": {
+      "cases": 4,
+      "exact_accuracy": 1.0,
+      "judge_accuracy": 1.0,
+      "mean_latency_ms": 29794.60875
+    }
+  },
+  "acceptance": {
+    "same_two_questions_all_paradigms_and_artifacts": true,
+    "png_and_pdf_used": true,
+    "chart_answers_absent_from_pdf_body_text": true,
+    "real_native_vision_calls": true,
+    "tool_selected_on_demand": true,
+    "real_tool_vision_calls": true,
+    "external_moonshot_judge": true,
+    "all_calls_checkpointed": true
+  }
+}

chapter4/multimodal-agent/validation/runs/20260729T185433Z-4_2-e028c9db/evidence.json

@@ -0,0 +1,466 @@
+{
+  "schema_version": "chapter3-evidence-v1",
+  "experiment": "4-2",
+  "run_id": "20260729T185433Z-4_2-e028c9db",
+  "provenance": {
+    "captured_at": "2026-07-29T18:54:33.359939+00:00",
+    "git_revision": "4a7f37cf278bd15948c409f14533017c4c7fbc29",
+    "python": "3.11.4 (main, Jul  5 2023, 08:40:20) [Clang 14.0.6 ]",
+    "platform": "macOS-26.3-arm64-arm-64bit",
+    "credential_presence": {
+      "ARK_API_KEY": true,
+      "MOONSHOT_API_KEY": true,
+      "OPENAI_API_KEY": true,
+      "GEMINI_API_KEY": true,
+      "SILICONFLOW_API_KEY": true
+    }
+  },
+  "status": "passed",
+  "providers": {
+    "vision_answerer": {
+      "provider": "Volcengine Ark",
+      "endpoint": "https://ark.cn-beijing.volces.com/api/v3",
+      "model": "doubao-seed-1-6-250615",
+      "seed": 37
+    },
+    "judge": {
+      "provider": "Moonshot",
+      "endpoint": "https://api.moonshot.cn/v1",
+      "model": "moonshot-v1-8k",
+      "seed": 37
+    }
+  },
+  "local_tools": {
+    "tesseract": "tesseract 5.5.2",
+    "pdftotext": "pdftotext version 25.06.0",
+    "pdftoppm": "pdftoppm version 25.06.0"
+  },
+  "artifacts": [
+    {
+      "kind": "png",
+      "source_path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_chart.png",
+      "source_sha256": "eaf8ac53fbc65a4e6ce0611d441cd12ddd2ed48ef8a727a9d6f22aef70605f39",
+      "visual_input": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_chart.png",
+      "visual_sha256": "eaf8ac53fbc65a4e6ce0611d441cd12ddd2ed48ef8a727a9d6f22aef70605f39",
+      "extracted_text": "Acme Corp Quarterly Revenue 2024\n200\n$180M\n175\n$150M\n150\nB 5 $120M\n&\n8 100 $95M\nG\n>\n2p\n50\n25\n0\nQl Q2 Q3 a4",
+      "extraction": {
+        "command": [
+          "tesseract",
+          "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_chart.png",
+          "stdout",
+          "--psm",
+          "6"
+        ],
+        "stderr": "",
+        "latency_ms": 203.405
+      }
+    },
+    {
+      "kind": "pdf",
+      "source_path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_report.pdf",
+      "source_sha256": "db4828a20a12ad70dab5c196d23e4fa407f75c4e3cff2bda22c401de65cb9c2e",
+      "visual_input": "/var/folders/0l/vk1w1b5n2fxfwdlz3f_w25_w0000gp/T/tmpqt5feern/sample_report_page.png",
+      "visual_sha256": "5688daad3a127e783fbb42e71d1d00bd8077d36a1ccf53e19f323d59b4e1d1e5",
+      "extracted_text": "Acme Corp 2024 Revenue Report\n\nThis internal report summarizes Acme Corp's revenue performance in 2024. Overall the year\nshowed healthy growth, with a mid-year dip followed by a strong recovery in the final quarter. The\nchart below breaks down revenue by quarter; management attributes the fourth-quarter surge to the\nlaunch of the new enterprise product line.",
+      "extraction": {
+        "command": [
+          "pdftotext",
+          "-layout",
+          "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_report.pdf",
+          "-"
+        ],
+        "stderr": "",
+        "latency_ms": 20.134
+      }
+    }
+  ],
+  "questions": [
+    {
+      "id": "highest",
+      "question": "Which quarter had the highest revenue, and what was the exact value?",
+      "expected": "Q4, $180M",
+      "required_patterns": [
+        "\\bQ4\\b",
+        "(?:\\$\\s*)?180\\s*M"
+      ]
+    },
+    {
+      "id": "lowest_gap",
+      "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+      "expected": "Q3, $95M; Q4 exceeded it by $85M",
+      "required_patterns": [
+        "\\bQ3\\b",
+        "(?:\\$\\s*)?95\\s*M",
+        "(?:\\$\\s*)?85\\s*M"
+      ]
+    }
+  ],
+  "results": [
+    {
+      "artifact": "png",
+      "question_id": "highest",
+      "question": "Which quarter had the highest revenue, and what was the exact value?",
+      "expected": "Q4, $180M",
+      "id": "png:native:highest",
+      "paradigm": "native-multimodal",
+      "answer": "The highest revenue was in Q4, with an exact value of $180M. Visual evidence: The Q4 bar is the tallest in the chart, explicitly labeled \"$180M\" and exceeding the revenue of Q1 ($120M), Q2 ($150M), and Q3 ($95M).\n\n**Answer:** Q4 had the highest revenue at $180M.",
+      "latency_ms": 8094.933,
+      "exact_correct": true,
+      "external_judge": {
+        "id": "png:native:highest",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer."
+      }
+    },
+    {
+      "artifact": "png",
+      "question_id": "highest",
+      "question": "Which quarter had the highest revenue, and what was the exact value?",
+      "expected": "Q4, $180M",
+      "id": "png:extract:highest",
+      "paradigm": "extract-to-text",
+      "answer": "Q1 had the highest revenue, with an exact value of $180M.",
+      "latency_ms": 7104.31,
+      "exact_correct": false,
+      "external_judge": {
+        "id": "png:extract:highest",
+        "correct": false,
+        "score": 0,
+        "reason": "The answer incorrectly states Q1 as the quarter with the highest revenue instead of Q4."
+      }
+    },
+    {
+      "artifact": "png",
+      "question_id": "highest",
+      "question": "Which quarter had the highest revenue, and what was the exact value?",
+      "expected": "Q4, $180M",
+      "id": "png:tool:highest",
+      "paradigm": "tool-on-demand",
+      "answer": "The highest revenue was in Q4 with an exact value of $180M.",
+      "latency_ms": 31752.022,
+      "exact_correct": true,
+      "tool_trace": {
+        "tool_selected": true,
+        "decision": {
+          "content": "",
+          "refusal": null,
+          "role": "assistant",
+          "annotations": null,
+          "audio": null,
+          "function_call": null,
+          "tool_calls": [
+            {
+              "id": "call_ep2ovjm4i87gakzdym8c7scs",
+              "function": {
+                "arguments": "{\"question\":\"What is the revenue for each quarter (Q1, Q2, Q3, Q4) in the Acme Corp Quarterly Revenue 2024 chart, and which quarter has the highest revenue with its exact value?\"}",
+                "name": "inspect_visual"
+              },
+              "type": "function"
+            }
+          ],
+          "reasoning_content": "我现在需要解决的问题是：“Which quarter had the highest revenue, and what was the exact value?” 也就是哪个季度的收入最高，具体数值是多少。首先，我得仔细看看用户提供的提取文本，看看有没有直接的信息。\n\n提取的文本是“Acme Corp Quarterly Revenue 2024”，然后下面有一些数字和符号，看起来像是图表的数据。具体的数字有200、180M、175、150M、150、B 5 $120M、&、8 100 $95M、G、>、2p、50、25、0，还有Ql Q2 Q3 a4。这里的Ql应该是Q1，a4可能是Q4的笔误。不过这些数字看起来比较混乱，可能是OCR识别错误或者格式问题。\n\n首先，我需要确定每个季度对应的收入数值。文本中出现了几个带有“M”的数值，比如$180M、$150M、$120M、$95M，这些可能是百万美元的收入。但问题在于这些数值和季度的对应关系不明确。比如，Q1、Q2、Q3、Q4分别对应哪个数值呢？\n\n提取的文本里有“Ql Q2 Q3 a4”，可能是Q1到Q4的顺序。但前面的数字排列看起来没有明显的顺序。比如200、175、150、100、50、25、0这些可能是纵轴的刻度，但具体对应哪个季度的数值呢？比如$180M可能对应某个季度，但文本中没有明确说明每个季度的具体数值。\n\n例如，文本中有“$180M”，但它旁边的数字是200和175，可能200是刻度，而180M是实际数值？或者175对应的是$150M？这里的空间关联不明确，因为文本提取可能没有正确捕捉到图表中的位置关系。比如，在柱状图或折线图中，每个季度的柱子或点会对应一个数值，但文本提取可能把这些数值和刻度混在一起了。\n\n因此，根据问题要求，当文本中没有明确建立精确的图表数值或空间关联时，需要调用inspect_visual工具。这里显然各个季度的收入数值没有明确对应，比如Q1的收入是多少，Q2是多少等，文本中的数字排列混乱，无法直接确定。所以必须通过视觉检查来确认每个季度对应的具体收入数值，从而找出最高的那个季度及其数值。\n\n所以，我需要调用inspect_visual工具，询问每个季度的收入数值，特别是最高的那个。具体的问题应该是：“What is the revenue for each quarter (Q1, Q2, Q3, Q4) in the Acme Corp Quarterly Revenue 2024 chart, and which quarter has the highest revenue with its exact value?” 这样就能明确获取所需的信息了。"
+        },
+        "executions": [
+          {
+            "tool_call_id": "call_ep2ovjm4i87gakzdym8c7scs",
+            "name": "inspect_visual",
+            "arguments": {
+              "question": "What is the revenue for each quarter (Q1, Q2, Q3, Q4) in the Acme Corp Quarterly Revenue 2024 chart, and which quarter has the highest revenue with its exact value?"
+            },
+            "result": "**Quarterly Revenues:**  \n- Q1: $120M (blue bar, labeled directly on the bar)  \n- Q2: $150M (green bar, labeled directly on the bar)  \n- Q3: $95M (red bar, labeled directly on the bar)  \n- Q4: $180M (purple bar, labeled directly on the bar)  \n\n\n**Highest Revenue:** Q4 with $180M (the tallest bar, explicitly labeled $180M)."
+          }
+        ]
+      },
+      "external_judge": {
+        "id": "png:tool:highest",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer."
+      }
+    },
+    {
+      "artifact": "png",
+      "question_id": "lowest_gap",
+      "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+      "expected": "Q3, $95M; Q4 exceeded it by $85M",
+      "id": "png:native:lowest_gap",
+      "paradigm": "native-multimodal",
+      "answer": "- **Lowest revenue quarter**: Q3  \n- **Value**: $95M  \n- **Q4 exceedance**: Q4 revenue ($180M) minus Q3 revenue ($95M) = $85M  \n\n\n**Visual evidence**: The red bar for Q3 is the shortest, labeled \"$95M,\" confirming it as the lowest. Q4’s purple bar is labeled \"$180M,\" the tallest. The vertical difference between Q4 and Q3 bars corresponds to $85M.",
+      "latency_ms": 10174.016,
+      "exact_correct": true,
+      "external_judge": {
+        "id": "png:native:lowest_gap",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer."
+      }
+    },
+    {
+      "artifact": "png",
+      "question_id": "lowest_gap",
+      "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+      "expected": "Q3, $95M; Q4 exceeded it by $85M",
+      "id": "png:extract:lowest_gap",
+      "paradigm": "extract-to-text",
+      "answer": "The lowest revenue was in Q1 with $95M, and Q4 exceeded it by $85M.",
+      "latency_ms": 29615.343,
+      "exact_correct": false,
+      "external_judge": {
+        "id": "png:extract:lowest_gap",
+        "correct": false,
+        "score": 0,
+        "reason": "The answer incorrectly states Q1 as the quarter with the lowest revenue instead of Q3."
+      }
+    },
+    {
+      "artifact": "png",
+      "question_id": "lowest_gap",
+      "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+      "expected": "Q3, $95M; Q4 exceeded it by $85M",
+      "id": "png:tool:lowest_gap",
+      "paradigm": "tool-on-demand",
+      "answer": "The lowest revenue occurred in Q3, with a value of $95M. Q4's revenue was $180M, exceeding Q3 by $85M ($180M - $95M).  \n\n**Answers:**  \n- Lowest revenue quarter: Q3  \n- Value: $95M  \n- Q4 exceeded it by: $85M",
+      "latency_ms": 35775.984,
+      "exact_correct": true,
+      "tool_trace": {
+        "tool_selected": true,
+        "decision": {
+          "content": "",
+          "refusal": null,
+          "role": "assistant",
+          "annotations": null,
+          "audio": null,
+          "function_call": null,
+          "tool_calls": [
+            {
+              "id": "call_d6od6lf1thf1cpwv1v6vjzje",
+              "function": {
+                "arguments": "{\"question\":\"What are the revenue values for Q1, Q2, Q3, and Q4 of Acme Corp in 2024? Which quarter has the lowest revenue, what is its value, and what is Q4's revenue?\"}",
+                "name": "inspect_visual"
+              },
+              "type": "function"
+            }
+          ],
+          "reasoning_content": "我现在需要解决的问题是：哪个季度的收入最低，其数值是多少，以及第四季度（Q4）比它高出多少？首先，我得仔细看看提供的提取文本，看看有没有直接的信息。\n\n提取的文本看起来有点混乱，可能是OCR识别的结果不太准确。首先，标题是“Acme Corp Quarterly Revenue 2024”，然后有一些数字和符号。比如“200”、“$180M”、“175”、“$150M”、“150”、“B 5 $120M”、“&”、“8 100 $95M”、“G”、“>”、“2p”、“50”、“25”、“0”，最后是“Ql Q2 Q3 a4”。这里的“Ql”应该是Q1，“a4”可能是Q4的笔误。\n\n首先，我需要确定每个季度对应的收入数值。通常季度是Q1到Q4，所以文本中的Ql、Q2、Q3、a4应该对应这四个季度。但问题在于，这些数字和符号如何与季度对应起来？比如，“$180M”、“$150M”、“$120M”、“$95M”看起来像是可能的收入数值，单位是百万美元。但这些数值是如何分配到各个季度的呢？\n\n提取的文本中没有明确说明每个季度对应的具体数值，只是列出了一些数字和符号。例如，“Ql Q2 Q3 a4”可能是在图表的底部，作为x轴的标签，而上面的数字可能是y轴的刻度或者数据点。但这里的文本提取似乎没有正确对应，比如“B 5 $120M”、“8 100 $95M”这样的组合，可能是识别错误，或者是图表中的某些元素被错误提取。\n\n在这种情况下，由于文本提取不明确，无法直接确定每个季度的收入数值，特别是空间关联（即哪个数值对应哪个季度）没有被文本明确建立。根据系统提示，当需要精确的图表数值或空间关联时，应该调用inspect_visual工具。\n\n因此，我需要调用inspect_visual工具，询问每个季度的具体收入数值，特别是最低的那个季度及其数值，以及Q4的数值，以便计算差额。具体来说，问题应该是：“请提供Acme Corp 2024年各季度（Q1、Q2、Q3、Q4）的收入数值，并指出最低收入的季度及其数值，以及Q4的收入数值。” 这样就能获取所需的精确数据，进而回答问题。"
+        },
+        "executions": [
+          {
+            "tool_call_id": "call_d6od6lf1thf1cpwv1v6vjzje",
+            "name": "inspect_visual",
+            "arguments": {
+              "question": "What are the revenue values for Q1, Q2, Q3, and Q4 of Acme Corp in 2024? Which quarter has the lowest revenue, what is its value, and what is Q4's revenue?"
+            },
+            "result": "**Revenue Values by Quarter (2024):**  \n- Q1: $120M (blue bar)  \n- Q2: $150M (green bar)  \n- Q3: $95M (red bar)  \n- Q4: $180M (purple bar)  \n\n\n**Lowest Revenue:** Q3, with $95M (smallest bar height, labeled directly).  \n\n\n**Q4 Revenue:** $180M (tallest bar, labeled directly).  \n\n\n*Visual Evidence:* The bar chart explicitly labels each quarter’s revenue above its respective bar, confirming the values. Q3’s red bar is the shortest, and Q4’s purple bar is the tallest."
+          }
+        ]
+      },
+      "external_judge": {
+        "id": "png:tool:lowest_gap",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer."
+      }
+    },
+    {
+      "artifact": "pdf",
+      "question_id": "highest",
+      "question": "Which quarter had the highest revenue, and what was the exact value?",
+      "expected": "Q4, $180M",
+      "id": "pdf:native:highest",
+      "paradigm": "native-multimodal",
+      "answer": "The fourth quarter (Q4) had the highest revenue, with an exact value of $180M.  \n\n**Visual evidence**: The Q4 bar in the \"Acme Corp Quarterly Revenue 2024\" bar graph is the tallest, explicitly labeled \"$180M,\" exceeding Q1 ($120M), Q2 ($150M), and Q3 ($95M).",
+      "latency_ms": 6289.762,
+      "exact_correct": true,
+      "external_judge": {
+        "id": "pdf:native:highest",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer."
+      }
+    },
+    {
+      "artifact": "pdf",
+      "question_id": "highest",
+      "question": "Which quarter had the highest revenue, and what was the exact value?",
+      "expected": "Q4, $180M",
+      "id": "pdf:extract:highest",
+      "paradigm": "extract-to-text",
+      "answer": "The fourth quarter had the highest revenue, but the exact value is unavailable.",
+      "latency_ms": 3313.326,
+      "exact_correct": false,
+      "external_judge": {
+        "id": "pdf:extract:highest",
+        "correct": false,
+        "score": 0,
+        "reason": "The answer states that the exact value is unavailable, which contradicts the reference answer that provides the exact value of $180M."
+      }
+    },
+    {
+      "artifact": "pdf",
+      "question_id": "highest",
+      "question": "Which quarter had the highest revenue, and what was the exact value?",
+      "expected": "Q4, $180M",
+      "id": "pdf:tool:highest",
+      "paradigm": "tool-on-demand",
+      "answer": "The highest revenue in 2024 was in the fourth quarter (Q4), with an exact value of $180M. This is supported by the visual bar chart labeled \"Acme Corp Quarterly Revenue 2024,\" which shows Q4 as the tallest bar explicitly marked with \"$180M,\" exceeding Q1 ($120M), Q2 ($150M), and Q3 ($95M). The report also notes a \"strong recovery in the final quarter\" attributed to the launch of the new enterprise product line, confirming Q4's surge.",
+      "latency_ms": 23111.276,
+      "exact_correct": true,
+      "tool_trace": {
+        "tool_selected": true,
+        "decision": {
+          "content": "",
+          "refusal": null,
+          "role": "assistant",
+          "annotations": null,
+          "audio": null,
+          "function_call": null,
+          "tool_calls": [
+            {
+              "id": "call_m638i2ydhoymnhz26ol5oad4",
+              "function": {
+                "arguments": "{\"question\":\"Which quarter had the highest revenue in 2024, and what was the exact revenue value for that quarter?\"}",
+                "name": "inspect_visual"
+              },
+              "type": "function"
+            }
+          ],
+          "reasoning_content": "我现在需要回答用户的问题：“Which quarter had the highest revenue, and what was the exact value?” 根据提供的提取文本，里面提到2024年整体表现健康增长，年中有所下滑，然后第四季度强劲复苏，管理层将第四季度的增长归因于新企业产品线的推出。不过，文本中并没有给出具体的季度收入数值，也没有明确说明哪个季度的收入最高，只是提到第四季度有增长。\n\n根据系统提示，当文本中没有明确的图表数值或空间关联时，需要调用inspect_visual工具。这里的问题需要知道具体的季度收入数值以及哪个季度最高，而文本中只提到第四季度有增长，但没有具体数字，所以必须通过视觉检查来获取这些信息。因此，我需要调用inspect_visual工具，询问哪个季度收入最高以及具体数值。"
+        },
+        "executions": [
+          {
+            "tool_call_id": "call_m638i2ydhoymnhz26ol5oad4",
+            "name": "inspect_visual",
+            "arguments": {
+              "question": "Which quarter had the highest revenue in 2024, and what was the exact revenue value for that quarter?"
+            },
+            "result": "The highest revenue in 2024 was in Q4, with an exact revenue value of $180M.  \n\n**Supporting visual evidence**: The bar chart labeled \"Acme Corp Quarterly Revenue 2024\" shows Q4 as the tallest bar, explicitly marked with \"$180M.\" This exceeds Q1 ($120M), Q2 ($150M), and Q3 ($95M). The report also notes a \"strong recovery in the final quarter\" attributed to the new enterprise product line, confirming Q4's surge."
+          }
+        ]
+      },
+      "external_judge": {
+        "id": "pdf:tool:highest",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q4 as the quarter with the highest revenue and provides the exact value of $180M, matching the reference answer and providing additional context."
+      }
+    },
+    {
+      "artifact": "pdf",
+      "question_id": "lowest_gap",
+      "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+      "expected": "Q3, $95M; Q4 exceeded it by $85M",
+      "id": "pdf:native:lowest_gap",
+      "paradigm": "native-multimodal",
+      "answer": "- **Lowest revenue quarter**: Q3  \n- **Value**: $95M  \n- **Q4 exceedance**: Q4 revenue ($180M) minus Q3 revenue ($95M) = $85M  \n\n\n**Visual evidence**: The bar graph shows Q3 with the shortest bar, labeled $95M. Q4’s bar is the tallest, labeled $180M. The vertical gap between Q3 and Q4 bars corresponds to $85M.",
+      "latency_ms": 7485.346,
+      "exact_correct": true,
+      "external_judge": {
+        "id": "pdf:native:lowest_gap",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer."
+      }
+    },
+    {
+      "artifact": "pdf",
+      "question_id": "lowest_gap",
+      "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+      "expected": "Q3, $95M; Q4 exceeded it by $85M",
+      "id": "pdf:extract:lowest_gap",
+      "paradigm": "extract-to-text",
+      "answer": "unavailable",
+      "latency_ms": 3436.729,
+      "exact_correct": false,
+      "external_judge": {
+        "id": "pdf:extract:lowest_gap",
+        "correct": false,
+        "score": 0,
+        "reason": "The answer states that the information is unavailable, which contradicts the reference answer that provides the exact values and the difference."
+      }
+    },
+    {
+      "artifact": "pdf",
+      "question_id": "lowest_gap",
+      "question": "Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?",
+      "expected": "Q3, $95M; Q4 exceeded it by $85M",
+      "id": "pdf:tool:lowest_gap",
+      "paradigm": "tool-on-demand",
+      "answer": "The lowest revenue quarter was Q3, with a value of $95M. Q4 revenue was $180M, exceeding Q3 by $85M. \n\n**Details**: The visual inspection confirms Q3 (red bar) as the lowest with $95M, while Q4 (purple bar) is the highest at $180M. The difference is calculated as $180M - $95M = $85M.",
+      "latency_ms": 28539.153,
+      "exact_correct": true,
+      "tool_trace": {
+        "tool_selected": true,
+        "decision": {
+          "content": "",
+          "refusal": null,
+          "role": "assistant",
+          "annotations": null,
+          "audio": null,
+          "function_call": null,
+          "tool_calls": [
+            {
+              "id": "call_df307c9zp27ddcazuecubeqh",
+              "function": {
+                "arguments": "{\"question\":\"Which quarter had the lowest revenue in 2024, what was its revenue value, what was the revenue value of Q4, and by how much did Q4 exceed the lowest quarter?\"}",
+                "name": "inspect_visual"
+              },
+              "type": "function"
+            }
+          ],
+          "reasoning_content": "我现在需要回答用户的问题：“Which quarter had the lowest revenue, what was its value, and by how much did Q4 exceed it?” 首先，我得仔细看看提供的提取文本，看看有没有相关的信息。\n\n根据提取的文本，里面提到“Overall the year showed healthy growth, with a mid-year dip followed by a strong recovery in the final quarter.” 这里提到了年中出现了一个低谷，然后第四季度强劲复苏。所以，最低的季度应该是年中的某个季度，可能是第二季度或者第三季度？不过文本里没有明确说是哪个季度，只是说“mid-year dip”，也就是年中低谷。年中通常指的是第二季度或者第三季度，但具体哪个呢？这时候可能需要视觉检查工具来确认图表中的具体数据。\n\n接下来，问题还问最低收入的具体数值，以及第四季度超过它多少。文本中没有提到任何具体的数值，只提到了第四季度的增长是由于新产品线的推出。所以，这些具体的数值必须通过查看图表才能获得。因此，我需要调用inspect_visual工具来获取这些信息。\n\n所以，我需要分步骤来处理：首先确定哪个季度是最低的，然后获取该季度的数值，再获取第四季度的数值，最后计算两者之间的差值。但根据文本，只能推断出年中低谷，具体哪个季度和数值都不知道，所以必须通过视觉检查来确认。\n\n因此，我应该调用inspect_visual工具，提出以下问题：“Which quarter had the lowest revenue in 2024, what was its revenue value, and what was the revenue value of Q4? Also, calculate the difference between Q4 and the lowest quarter.” 这样就能一次性获取所有需要的信息，包括最低季度、其数值、Q4的数值以及两者的差值。\n\n不过，可能需要更明确的问题结构。比如，先问最低的季度，再问该季度的数值，然后问Q4的数值，最后计算差值。但工具可能可以一次处理多个问题，所以综合成一个问题应该可以。或者分开调用？不过用户允许并行调用，所以可能可以一次问多个问题。不过根据工具的描述，每次调用只能有一个question参数，所以可能需要分多次调用？或者在一个question里包含多个子问题？\n\n根据工具的parameters，question是一个字符串，所以可能可以在一个问题里包含多个询问点。例如：“Which quarter had the lowest revenue in 2024? What was its revenue value? What was the revenue value of Q4? By how much did Q4 exceed the lowest quarter?” 这样工具可能会返回所有这些信息。\n\n因此，我应该构造一个包含所有这些问题的查询，调用inspect_visual工具，以获取所需的数据。这样就能回答用户的问题了。"
+        },
+        "executions": [
+          {
+            "tool_call_id": "call_df307c9zp27ddcazuecubeqh",
+            "name": "inspect_visual",
+            "arguments": {
+              "question": "Which quarter had the lowest revenue in 2024, what was its revenue value, what was the revenue value of Q4, and by how much did Q4 exceed the lowest quarter?"
+            },
+            "result": "- **Lowest revenue quarter**: Q3  \n- **Lowest revenue value**: $95M  \n- **Q4 revenue value**: $180M  \n- **Q4 exceeds lowest quarter by**: $85M  \n\n\n**Visual evidence**: The bar chart shows Q3 (red bar) with the shortest height, labeled $95M. Q4 (purple bar) is the tallest, labeled $180M. The difference is $180M - $95M = $85M."
+          }
+        ]
+      },
+      "external_judge": {
+        "id": "pdf:tool:lowest_gap",
+        "correct": true,
+        "score": 1,
+        "reason": "The answer correctly identifies Q3 as the quarter with the lowest revenue, provides the exact value of $95M, and correctly calculates the difference of $85M that Q4 exceeded it by, matching the reference answer."
+      }
+    }
+  ],
+  "summary": {
+    "native-multimodal": {
+      "cases": 4,
+      "exact_accuracy": 1.0,
+      "judge_accuracy": 1.0,
+      "mean_latency_ms": 8011.01425
+    },
+    "extract-to-text": {
+      "cases": 4,
+      "exact_accuracy": 0.0,
+      "judge_accuracy": 0.0,
+      "mean_latency_ms": 10867.427
+    },
+    "tool-on-demand": {
+      "cases": 4,
+      "exact_accuracy": 1.0,
+      "judge_accuracy": 1.0,
+      "mean_latency_ms": 29794.60875
+    }
+  },
+  "acceptance": {
+    "same_two_questions_all_paradigms_and_artifacts": true,
+    "png_and_pdf_used": true,
+    "chart_answers_absent_from_pdf_body_text": true,
+    "real_native_vision_calls": true,
+    "tool_selected_on_demand": true,
+    "real_tool_vision_calls": true,
+    "external_moonshot_judge": true,
+    "all_calls_checkpointed": true
+  },
+  "checkpoint_files": [
+    "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/validation/checkpoints/20260729T185104Z-ark.json",
+    "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/validation/checkpoints/20260729T185104Z-judge.json"
+  ]
+}

chapter4/multimodal-agent/validation/runs/20260729T185433Z-4_2-e028c9db/manifest.json

@@ -0,0 +1,64 @@
+{
+  "schema_version": "chapter3-evidence-v1",
+  "experiment": "4-2",
+  "run_id": "20260729T185433Z-4_2-e028c9db",
+  "created_at": "2026-07-29T18:54:33.367744+00:00",
+  "status": "passed",
+  "run_dir": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/validation/runs/20260729T185433Z-4_2-e028c9db",
+  "artifacts": {
+    "evidence.json": "024358e7b58266eedbedda3cc880a61462c119a89c58e475f4c007dbcb8ff144",
+    "receipts.json": "2aa961a1f7527417dd0d25ac6e542aaad9dc2ffaa7b40a4a450cb5fffe86ceac"
+  },
+  "inputs": [
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/campaign.py",
+      "sha256": "6f8162a0e124cef1acada4fbfadc6a1427aa487be24670aaceba2d668820ee3c",
+      "bytes": 16104
+    },
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/create_sample.py",
+      "sha256": "1f4fcd0e119a76a74c06744a585d31bc4bd4f755f813f8b2518f77127eaa1e72",
+      "bytes": 4846
+    },
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_chart.png",
+      "sha256": "eaf8ac53fbc65a4e6ce0611d441cd12ddd2ed48ef8a727a9d6f22aef70605f39",
+      "bytes": 38417
+    },
+    {
+      "path": "/Users/boj/book/ai-agent-book/chapter4/multimodal-agent/test_files/sample_report.pdf",
+      "sha256": "db4828a20a12ad70dab5c196d23e4fa407f75c4e3cff2bda22c401de65cb9c2e",
+      "bytes": 46368
+    }
+  ],
+  "summary": {
+    "native-multimodal": {
+      "cases": 4,
+      "exact_accuracy": 1.0,
+      "judge_accuracy": 1.0,
+      "mean_latency_ms": 8011.01425
+    },
+    "extract-to-text": {
+      "cases": 4,
+      "exact_accuracy": 0.0,
+      "judge_accuracy": 0.0,
+      "mean_latency_ms": 10867.427
+    },
+    "tool-on-demand": {
+      "cases": 4,
+      "exact_accuracy": 1.0,
+      "judge_accuracy": 1.0,
+      "mean_latency_ms": 29794.60875
+    }
+  },
+  "acceptance": {
+    "same_two_questions_all_paradigms_and_artifacts": true,
+    "png_and_pdf_used": true,
+    "chart_answers_absent_from_pdf_body_text": true,
+    "real_native_vision_calls": true,
+    "tool_selected_on_demand": true,
+    "real_tool_vision_calls": true,
+    "external_moonshot_judge": true,
+    "all_calls_checkpointed": true
+  }
+}