ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter7/user-memory-system-evaluation/README.md

@@ -0,0 +1,219 @@
+# Experiments 7-4 and 7-11: end-to-end user-memory evaluation
+
+This companion runs memory systems. It does not score canned response files.
+It reuses the 60 cases in `chapter3/user-memory-evaluation/test_cases` and
+records an API-backed trajectory for every `(case, configuration)` cell.
+
+← [Chapter 7 index](../README.md) · [Book acceptance criteria](../../book/chapter7.md)
+
+## What is implemented
+
+### Experiment 7-4: Advanced JSON Cards vs RAG vs hybrid
+
+For every one of the same 60 cases, the runner independently builds and runs:
+
+| System | Ingestion and answering path | Steps/tools |
+| --- | --- | --- |
+| Advanced JSON Cards | An LLM extracts structured cards containing provenance, person/relationship, exact facts, temporal status, and ambiguity; all cards stay in the answer context. | One answer step, zero retrieval tools |
+| RAG | Raw conversations are split on complete turns, embedded into a dense index, searched through an actual `search_memory` tool call, optionally reranked, then answered from top-5 chunks. | Forced retrieval plus answer |
+| Hybrid | Only cards explicitly classified `memory_tier: core` stay resident; supporting/episodic facts remain in raw conversations while the main Agent decides whether to call `search_memory`. | One or two steps; tool use is observed, not hard-coded |
+
+The JSON report records success/reward, rubric dimensions, hallucination veto,
+steps, tool calls, latency, input/output tokens, cost and price-coverage gaps.
+`success` requires at least `good` (3/4) on precision, recall and reasoning plus
+no hallucination veto; `reward` still preserves partial credit.
+`failure_boundaries` lists failed cases and per-dimension weaknesses for each
+system/layer, plus a paired hybrid-synergy/regression analysis.
+
+### Experiment 7-11: full component matrix
+
+`default_config.yaml` sweeps all three selection points from the book:
+
+- embeddings: BGE-M3, OpenAI, and an independently hosted Mistral control,
+  plus a documented Qwen3 substitution for the unreachable Doubao embedding
+  (see "Backend substitutions" below);
+- rerankers: no-reranker baseline, a Doubao semantic reranker (documented
+  substitution for the unreachable BGE cross-encoder), and the Kimi semantic
+  reranker;
+- main models: Kimi and Ark/Doubao under an identical retrieval contract.
+
+### Backend substitutions (2026-07-31)
+
+Acceptance is tied to equivalent providers/models, not to one vendor's
+official API. Every substitution is recorded in `default_config.yaml` and in
+the sanitized receipts `results/candidate_backend_probes_20260731.json` and
+`results/full_matrix_backend_readiness_20260731.json`:
+
+- SiliconFlow's key is valid but the account balance is 0 (HTTP 402), so
+  `bge-m3` runs the identical `baai/bge-m3` model via OpenRouter.
+- The direct OpenAI account has no credits (HTTP 429), so `openai-small`
+  runs the identical `openai/text-embedding-3-small` via OpenRouter.
+- Ark embeddings require a console-provisioned endpoint id and every public
+  Doubao embedding model name returns 404 on this account, so the Doubao
+  embedding slot is honestly replaced by `qwen/qwen3-embedding-8b` via
+  OpenRouter (the closest Chinese-provider multilingual embedding).
+- No cross-encoder reranker is reachable (SiliconFlow balance 0; DashScope
+  gte-rerank returns 403 AccessDenied with this international key), so the
+  BGE cross-encoder slot is honestly replaced by `doubao-semantic`, a second
+  LLM reranker on the Doubao chat model. The matrix therefore compares
+  none / Doubao-LLM / Kimi-LLM reranking; no cross-encoder is claimed.
+
+A source-aware retrieval judge selects the relevant chunk IDs before the matrix
+run. Each cell is then measured with hit@5, recall@5 and MRR, as well as task
+success, rubric score, steps, tool calls, latency and cost. The report does not
+rank components in isolation: `interaction_analysis` calculates reranker value
+conditional on embedding and main model, flags observed reranker redundancy,
+and measures whether stronger main models succeed despite incomplete retrieval.
+
+Embedding/reranker quality is also measured with an identical fixed user-query
+benchmark in every cell (`fixed_query_*`), avoiding main-model query wording as
+a confound. The production Agent trajectory is measured separately: retrieval
+is mandatory, but the main model may make up to three follow-up searches, so
+steps/tool calls are real efficiency signals instead of constants.
+
+Provider failures become explicit `status: error` matrix records and never count
+as task failures. This prevents an unavailable account or endpoint from silently
+changing a quality comparison.
+
+The report has a machine-readable `run_scope`. A run is marked `full` only when
+all 60 distinct case IDs and all configured cells completed. Filtered evidence
+is always marked `smoke`; a 60-case invocation with provider errors is marked
+`incomplete-full-suite`.
+
+## Experiment 7-3 prerequisite
+
+The shared judge in [`chapter3/user-memory-evaluation`](../../chapter3/user-memory-evaluation/)
+is now the structured Experiment 7-3 judge. It sees the authoritative source and
+returns four grades for precision, recall, reasoning, and proactivity, with
+evidence and boundary cases. A separate hallucination result is a hard veto.
+The runner here uses that judge for 7-4 and 7-11 task success.
+
+The completed 7-4 campaign also provides the full execution evidence for 7-3:
+all 60 distinct cases across three systems produced 180/180 real structured
+judgments. [`results/full_7_3_structured_rubric_evidence.json`](results/full_7_3_structured_rubric_evidence.json)
+validates every saved record against the four-dimension contract and independent
+hallucination veto, and content-hashes the immutable source report. It is built
+by `python build_73_evidence.py`; the derivation performs no model calls and
+does not add or change any score.
+
+## Install and configure
+
+```bash
+cd chapter7/user-memory-system-evaluation
+python -m pip install -r requirements.txt
+cp env.example .env
+```
+
+Credentials are read only from environment variables; reports never contain
+keys. `default_config.yaml` is the full book matrix. All matrix components
+carry dated list prices so `unpriced_tokens` stays zero; the report exposes
+`unpriced_tokens` so incomplete cost accounting cannot look like a zero-cost
+system.
+
+## Run
+
+The default is all 60 cases:
+
+```bash
+python experiment.py 7-4 --config default_config.yaml \
+  --output results/experiment_7_4.json
+
+python experiment.py 7-11 --config default_config.yaml \
+  --output results/experiment_7_11.json
+```
+
+Use filters only for smoke tests:
+
+```bash
+python experiment.py 7-4 --config live_config.yaml \
+  --test-id layer1_01_bank_account \
+  --output results/live_7_4_layer1.json
+
+python experiment.py 7-11 --config live_config.yaml \
+  --test-id layer1_01_bank_account \
+  --output results/live_7_11_matrix_layer1.json
+```
+
+Restart-safe complete campaigns:
+
+```bash
+python run_full.py 7-4 --config live_config.yaml --workers 4 \
+  --output results/full_7_4_60_cases.json
+
+python run_full.py 7-11 --config default_config.yaml --workers 4 \
+  --readiness results/full_matrix_backend_readiness.json \
+  --output results/full_7_11_60_case_matrix.json
+```
+
+`run_full.py` writes one case checkpoint before counting it, resumes valid
+checkpoints, and merges only direct records. A readiness file avoids repeatedly
+calling a provider already proven unavailable while still emitting every blocked
+matrix cell as `status: error`.
+
+`live_config.yaml` is a known-working development-account subset. It uses real
+Mistral/Codestral embeddings, no-reranker and Kimi reranker, and Kimi/Doubao main
+models. It does not replace the full BGE/OpenAI/Doubao matrix.
+
+Probe the full configuration without running 60 cases:
+
+```bash
+python probe_backends.py --config default_config.yaml \
+  --output results/full_matrix_backend_readiness.json
+```
+
+The probe calls the actual configured chat, embedding, and reranking paths and
+stores sanitized status/error evidence. Keys are never written.
+
+## Tests and checked-in live evidence
+
+```bash
+pytest -q ../../chapter3/user-memory-evaluation/test_structured_rubric.py test_experiment.py
+```
+
+- `results/live_7_4_core_hybrid_layer1.json`: three complete layer-1 7-4 trajectories
+  using the exact core-card hybrid path.
+- `results/full_7_4_60_cases_costed.json`: canonical completed Experiment 7-4
+  campaign—60 distinct cases × three systems, 180/180 real trajectories, zero
+  trajectory errors, `validation_scope: full`, and complete native-currency cost
+  coverage. Its top-level and completion status are both `complete`.
+- `results/live_7_11_matrix_layer1.json`: current-code live factorial 7-11 smoke
+  (generated by the command above when present).
+- `../../chapter3/user-memory-evaluation/results/live_7_3_layer1.json`: live Kimi structured-rubric result.
+- `../../chapter3/user-memory-evaluation/results/live_7_3_hallucination_veto.json`:
+  live Kimi proof that one unsupported number forces reward to zero.
+- `results/full_matrix_backend_readiness.json`: sanitized full-matrix endpoint probe.
+- `results/full_matrix_backend_readiness_20260731.json`: sanitized 9/9 readiness
+  probe under the documented substitutions; `results/candidate_backend_probes_20260731.json`
+  keeps the per-candidate rejection receipts (SiliconFlow 402 balance, OpenAI 429,
+  Ark embedding 404s, DashScope rerank 403) that justify each substitution.
+
+These evidence files contain synthetic benchmark answers, metrics and model
+names, but no credentials or complete source conversations. Experiment 7-4 is
+complete only through the canonical full report named above; the `live_*` files
+remain smoke evidence and must not be substituted for it.
+
+Experiment 7-11 is **complete**: the full 4×3×2×60 matrix campaign finished with
+1,440/1,440 real trajectories, zero error records, and zero unpriced usage in
+`results/full_7_11_60_case_matrix.json` (top-level and completion status both
+`complete`), executed under the documented backend substitutions above
+(`results/full_matrix_backend_readiness_20260731.json`).
+`validation/verify_full_matrix_20260731.py` independently rechecks case/cell
+coverage, trajectory cleanliness, metric finiteness, pricing coverage, and the
+interaction analysis (ALL CHECKS PASSED).
+None of the earlier blockers changed the completed 7-4 status.
+
+## 中文说明
+
+本目录对应实验 7-4 与 7-11，实际构建并运行三种记忆系统及组件矩阵，不再对预先写好的
+回答文件打分。默认读取第三章同一套 60 个测试用例，逐条记录任务成功率、步数、工具调用、
+延迟、token、成本覆盖、top-5 检索指标和结构化 Rubric。`default_config.yaml` 是正文要求的
+BGE-M3 / OpenAI / 豆包嵌入、含无 reranker 基线、以及多主模型的完整矩阵；
+`live_config.yaml` 只是已验证账号的真实 API 冒烟子集。实验 7-3 的五维 Rubric（四个评分维度
++ 幻觉否决）位于第三章共用评估框架，并由本目录直接复用。
+
+当前状态必须按实验分别读取：实验 7-4 已由
+`results/full_7_4_60_cases_costed.json` 完成 60 用例 × 3 系统共 180/180 条真实轨迹和完整成本核算；
+实验 7-11 的 4×3×2×60 全矩阵活动已完成：`results/full_7_11_60_case_matrix.json` 收录 60 用例 × 24 单元
+共 1,440/1,440 条真实轨迹，零错误、零未定价用量，检索/任务指标与交互分析完整（顶层与 completion
+状态均为 `complete`），并由 `validation/verify_full_matrix_20260731.py` 独立复核通过。
+矩阵在后端就绪度 9/9 的如实记录替代方案下执行（见上文“Backend substitutions”）。

chapter7/user-memory-system-evaluation/build_73_evidence.py

@@ -0,0 +1,149 @@
+#!/usr/bin/env python3
+"""Derive canonical Experiment 7-3 evidence from the completed 7-4 campaign.
+
+The completed 7-4 report ran the Experiment 7-3 judge on every one of its
+60 cases and three memory systems.  This validator creates a small, auditable
+index without changing, adding, or re-judging any paid API trajectory.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+from collections import Counter, defaultdict
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+REQUIRED_DIMENSIONS = {"precision", "recall", "reasoning", "proactivity"}
+
+
+def sha256(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--source",
+        type=Path,
+        default=Path(__file__).with_name("results") / "full_7_4_60_cases_costed.json",
+    )
+    parser.add_argument(
+        "--output",
+        type=Path,
+        default=Path(__file__).with_name("results") / "full_7_3_structured_rubric_evidence.json",
+    )
+    args = parser.parse_args()
+
+    source = json.loads(args.source.read_text(encoding="utf-8"))
+    records = source.get("records", [])
+    valid = []
+    errors = []
+    by_layer_system = defaultdict(lambda: Counter(records=0, passed=0, vetoes=0))
+    for index, row in enumerate(records):
+        dimensions = row.get("rubric_details") or {}
+        numeric = row.get("rubric_dimensions") or {}
+        hallucination = row.get("hallucination_detail")
+        problems = []
+        if row.get("status") != "ok":
+            problems.append(f"status={row.get('status')!r}")
+        if set(dimensions) != REQUIRED_DIMENSIONS:
+            problems.append(f"rubric_details={sorted(dimensions)}")
+        if set(numeric) != REQUIRED_DIMENSIONS:
+            problems.append(f"rubric_dimensions={sorted(numeric)}")
+        if any(not 1 <= int(value) <= 4 for value in numeric.values()):
+            problems.append("rubric score outside 1..4")
+        if not isinstance(hallucination, dict) or "detected" not in hallucination:
+            problems.append("missing hallucination verdict")
+        for name, detail in dimensions.items():
+            # A concise direct answer can legitimately have no affirmative
+            # proactivity evidence.  In that boundary case the judge must name
+            # the applied boundary explicitly instead of inventing evidence.
+            if (
+                not isinstance(detail, dict)
+                or not detail.get("reasoning")
+                or not (detail.get("evidence") or detail.get("boundary_case"))
+            ):
+                problems.append(f"{name} lacks reasoning and evidence/boundary")
+        if problems:
+            errors.append({
+                "record_index": index,
+                "test_id": row.get("test_id"),
+                "system": row.get("system"),
+                "problems": problems,
+            })
+            continue
+        valid.append(row)
+        bucket = by_layer_system[(row["layer"], row["system"])]
+        bucket["records"] += 1
+        bucket["passed"] += int(bool(row.get("success")))
+        bucket["vetoes"] += int(bool(row.get("hallucination_veto")))
+
+    distinct_cases = sorted({row.get("test_id") for row in valid})
+    systems = sorted({row.get("system") for row in valid})
+    layers = sorted({row.get("layer") for row in valid})
+    complete = (
+        not errors
+        and len(records) == 180
+        and len(valid) == 180
+        and len(distinct_cases) == 60
+        and len(systems) == 3
+        and layers == ["layer1", "layer2", "layer3"]
+        and all(counter["records"] == 20 for counter in by_layer_system.values())
+        and len(by_layer_system) == 9
+    )
+    report = {
+        "schema_version": "1.0",
+        "experiment": "7-3",
+        "generated_at_utc": datetime.now(timezone.utc).isoformat(),
+        "evidence_lineage": {
+            "source_file": str(args.source),
+            "source_sha256": sha256(args.source),
+            "source_experiment": source.get("experiment"),
+            "source_generated_at_utc": source.get("generated_at_utc"),
+            "transformation": (
+                "Validation/index only: no API records, answers, scores, or verdicts were added, "
+                "removed, or changed."
+            ),
+        },
+        "rubric_contract": {
+            "dimensions": sorted(REQUIRED_DIMENSIONS),
+            "scale": "1..4 with concrete reasoning and cited evidence",
+            "hallucination": "independent hard veto",
+        },
+        "run_scope": {
+            "distinct_test_cases": len(distinct_cases),
+            "layers": layers,
+            "systems": systems,
+            "records_expected": 180,
+            "records_validated": len(valid),
+            "all_60_cases_covered": len(distinct_cases) == 60,
+            "validation_scope": "full" if complete else "incomplete",
+        },
+        "summary": [
+            {"layer": layer, "system": system, **dict(counter)}
+            for (layer, system), counter in sorted(by_layer_system.items())
+        ],
+        "errors": errors,
+        "status": "complete" if complete else "incomplete",
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
+    print(json.dumps({
+        "status": report["status"],
+        "records_validated": len(valid),
+        "distinct_cases": len(distinct_cases),
+        "errors": len(errors),
+        "output": str(args.output),
+    }, ensure_ascii=False))
+    return 0 if complete else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter7/user-memory-system-evaluation/calibration_summary.py

@@ -0,0 +1,61 @@
+#!/usr/bin/env python3
+"""Summarize calibration output: per-case calls/tokens/cost, projected to 60 cases."""
+
+import json
+import sys
+from collections import defaultdict
+
+path = sys.argv[1]
+data = json.load(open(path))
+records = data["records"]
+ok = [r for r in records if r["status"] == "ok"]
+err = [r for r in records if r["status"] == "error"]
+print(f"records={len(records)} ok={len(ok)} error={len(err)}")
+
+tokens_by_cell_component = defaultdict(int)
+chat_in = defaultdict(int)
+chat_out = defaultdict(int)
+costs = defaultdict(float)
+unpriced_tokens = 0
+unpriced_requests = 0
+latencies = []
+
+for r in ok:
+    latencies.append(r["latency_ms"])
+    unpriced_tokens += r["unpriced_tokens"] + r["fixed_query_unpriced_tokens"]
+    unpriced_requests += r["unpriced_requests"] + r["fixed_query_unpriced_requests"]
+    for cur, amt in r.get("cost_by_currency", {}).items():
+        costs[cur] += amt
+    for cur, amt in r.get("fixed_query_retrieval_cost_by_currency", {}).items():
+        costs[cur] += amt
+    # main+reranker+judge tokens are merged in input/output tokens;
+    # fixed-query tokens are separate.
+    chat_in[r["main_model"]] += r["input_tokens"]
+    chat_out[r["main_model"]] += r["output_tokens"]
+
+print("\nPer-case totals (one case = 24 cells + 12 fixed-query benchmarks):")
+print(f"  primary input tokens by main model: {dict(chat_in)}")
+print(f"  primary output tokens by main model: {dict(chat_out)}")
+print(f"  fixed-query tokens: {sum(r['fixed_query_input_tokens'] + r['fixed_query_output_tokens'] for r in ok)}")
+print(f"  cost by currency: {dict(costs)}")
+print(f"  unpriced tokens: {unpriced_tokens}, unpriced requests: {unpriced_requests}")
+print(f"  latency_ms sum over records: {sum(latencies):.0f} "
+      f"(serial per-cell latency; per-case wall clock differs)")
+
+print("\nProjected x60 cases:")
+for cur, amt in costs.items():
+    print(f"  {cur}: {amt * 60:.2f}")
+print(f"  primary input tokens: {sum(chat_in.values()) * 60:,}")
+print(f"  primary output tokens: {sum(chat_out.values()) * 60:,}")
+
+# steps/tool calls distribution
+import statistics
+steps = [r["steps"] for r in ok]
+tools = [r["tool_calls"] for r in ok]
+print(f"\nsteps: mean={statistics.fmean(steps):.2f} max={max(steps)}; "
+      f"tool_calls: mean={statistics.fmean(tools):.2f} max={max(tools)}")
+by_rr = defaultdict(list)
+for r in ok:
+    by_rr[(r["reranker"], r["main_model"])].append(r["latency_ms"])
+for k, v in sorted(by_rr.items(), key=str):
+    print(f"  {k}: n={len(v)} mean latency {statistics.fmean(v)/1000:.1f}s")

chapter7/user-memory-system-evaluation/default_config.yaml

@@ -0,0 +1,143 @@
+# Full book matrix. Prices are dated provider list prices in the native
+# published currency. Never add an FX conversion without its own dated source.
+#
+# Backend substitutions recorded 2026-07-31 (acceptance is tied to equivalent
+# providers/models, not to one vendor's official API; every substitution is
+# documented here and in README.md):
+# - SiliconFlow is unfunded (HTTP 402 balance=0, key itself valid), so the
+#   BGE-M3 embedding and the BGE cross-encoder reranker cannot run there.
+#   BGE-M3 now runs as the identical model `baai/bge-m3` via OpenRouter.
+#   No reachable cross-encoder reranker remains (DashScope gte-rerank returns
+#   403 AccessDenied on this account, and the mainland endpoint rejects the
+#   international key), so the cross-encoder matrix slot is honestly replaced
+#   by a second LLM reranker, `doubao-semantic`, keeping three distinct
+#   reranking strategies (none / Kimi-LLM / Doubao-LLM).
+# - The direct OpenAI account has zero credits (HTTP 429), so the OpenAI
+#   embedding runs as the identical model `openai/text-embedding-3-small`
+#   via OpenRouter.
+# - Doubao embeddings on Ark require a console-provisioned endpoint id; every
+#   public model name returns 404 InvalidEndpointOrModel on this account.
+#   The slot is honestly replaced by `qwen/qwen3-embedding-8b` via OpenRouter
+#   (Alibaba Qwen3-Embedding-8B, the closest Chinese-provider multilingual
+#   embedding substitute).
+chat_models:
+  kimi:
+    model: kimi-k2.5
+    base_url: https://api.moonshot.cn/v1
+    api_key_env: KIMI_API_KEY
+    disable_thinking: true
+    temperature: 0.6
+    pricing:
+      currency: CNY
+      as_of_date: "2026-07-29"
+      source_url: https://platform.kimi.com/docs/pricing/chat-k25
+      input_per_million: 4.00
+      cached_input_per_million: 0.70
+      output_per_million: 21.00
+      source_note: Published Kimi K2.5 list price; input rate is cache-miss/uncached.
+  doubao:
+    model: doubao-seed-1-6-250615
+    base_url: https://ark.cn-beijing.volces.com/api/v3
+    api_key_env: ARK_API_KEY
+    pricing:
+      currency: CNY
+      as_of_date: "2026-07-31"
+      source_url: https://www.volcengine.com/docs/82379/1544106
+      input_per_million: 0.80
+      output_per_million: 8.00
+      source_note: Published Doubao-Seed-1.6 list price for the 0-32K input-length
+        range (input 0.8 CNY/M, output 8 CNY/M). All matrix prompts are below 32K.
+  openai-mini:
+    model: gpt-4.1-mini
+    base_url: https://api.openai.com/v1
+    api_key_env: OPENAI_API_KEY
+
+embeddings:
+  bge-m3:
+    # Same BAAI/bge-m3 model as the book matrix; provider substituted
+    # SiliconFlow -> OpenRouter on 2026-07-31 (SiliconFlow balance is 0).
+    model: baai/bge-m3
+    base_url: https://openrouter.ai/api/v1
+    api_key_env: OPENROUTER_API_KEY
+    pricing:
+      currency: USD
+      as_of_date: "2026-07-31"
+      source_url: https://openrouter.ai/baai/bge-m3
+      input_per_million: 0.01
+      source_note: OpenRouter catalog price for baai/bge-m3; matches the per-token
+        cost returned by the /api/v1/embeddings/models listing and live usage.
+  openai-small:
+    # Same text-embedding-3-small model as the book matrix; provider substituted
+    # direct OpenAI -> OpenRouter on 2026-07-31 (OpenAI account has no credits).
+    model: openai/text-embedding-3-small
+    base_url: https://openrouter.ai/api/v1
+    api_key_env: OPENROUTER_API_KEY
+    pricing:
+      currency: USD
+      as_of_date: "2026-07-31"
+      source_url: https://openrouter.ai/openai/text-embedding-3-small
+      input_per_million: 0.02
+      source_note: OpenRouter catalog price, identical to the published
+        text-embedding-3-small list price.
+  qwen3:
+    # Substitution for the unavailable Doubao embedding slot (Ark embeddings
+    # need a provisioned endpoint id; all public model names 404 on this
+    # account). Qwen3-Embedding-8B is the closest Chinese-provider multilingual
+    # embedding reachable via OpenRouter.
+    model: qwen/qwen3-embedding-8b
+    base_url: https://openrouter.ai/api/v1
+    api_key_env: OPENROUTER_API_KEY
+    pricing:
+      currency: USD
+      as_of_date: "2026-07-31"
+      source_url: https://openrouter.ai/qwen/qwen3-embedding-8b
+      input_per_million: 0.01
+      source_note: OpenRouter catalog price for qwen/qwen3-embedding-8b.
+  mistral:
+    model: mistral-embed
+    base_url: https://api.mistral.ai/v1
+    api_key_env: MISTRAL_API_KEY
+    pricing:
+      currency: USD
+      as_of_date: "2026-07-29"
+      source_url: https://mistral.ai/pricing/api/
+      input_per_million: 0.10
+      source_note: Published Mistral Embed list price.
+
+rerankers:
+  none:
+    type: none
+  doubao-semantic:
+    # Substitution for the unavailable BGE cross-encoder slot (SiliconFlow
+    # balance 0; DashScope gte-rerank 403 AccessDenied on this account).
+    # A second LLM reranker on a different model keeps three distinct
+    # reranking strategies without pretending a cross-encoder ran.
+    type: llm
+    chat_model: doubao
+  kimi-semantic:
+    type: llm
+    chat_model: kimi
+
+judge:
+  evaluator: kimi
+  model: kimi-k2.5
+
+experiment_7_4:
+  main_model: kimi
+  embedding: mistral
+  reranker: kimi-semantic
+  rounds_per_chunk: 6
+  overlap: 2
+
+experiment_7_11:
+  # The named adapters cover the BGE-M3/OpenAI/Mistral comparison from the
+  # book plus the documented Qwen3 substitution for the unreachable Doubao
+  # embedding endpoint; rerankers compare none / Doubao-LLM / Kimi-LLM after
+  # the documented cross-encoder substitution.
+  embeddings: [bge-m3, openai-small, qwen3, mistral]
+  rerankers: [none, doubao-semantic, kimi-semantic]
+  main_models: [kimi, doubao]
+  retrieval_judge_model: kimi
+  max_search_rounds: 3
+  rounds_per_chunk: 6
+  overlap: 2

chapter7/user-memory-system-evaluation/enrich_rubrics.py

@@ -0,0 +1,104 @@
+#!/usr/bin/env python3
+"""Add complete 7-3 rubric evidence to saved 7-4/7-11 case checkpoints."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Any, Dict
+
+from experiment import EVAL_DIR, UserMemoryEvaluationFramework
+from evaluator import LLMEvaluator
+
+
+REQUIRED_DIMENSIONS = {"precision", "recall", "reasoning", "proactivity"}
+
+
+def enrich_case(path: Path, evaluator_type: str, model: str, test_cases_dir: Path) -> Dict[str, Any]:
+    framework = UserMemoryEvaluationFramework(str(test_cases_dir))
+    evaluator = LLMEvaluator(evaluator_type, model=model)
+    payload = json.loads(path.read_text(encoding="utf-8"))
+    updated = 0
+    errors = []
+    for row in payload.get("records", []):
+        if row.get("status") != "ok":
+            continue
+        if set(row.get("rubric_details", {})) == REQUIRED_DIMENSIONS and row.get("hallucination_detail"):
+            continue
+        test_case = framework.get_test_case(row["test_id"])
+        result = evaluator.evaluate(test_case, row["answer"])
+        if set(result.dimensions) != REQUIRED_DIMENSIONS or result.hallucination is None:
+            errors.append({"system": row["system"], "reason": result.reasoning})
+            continue
+        row["reward"] = result.reward
+        row["success"] = bool(result.passed)
+        row["rubric_dimensions"] = {name: value.score for name, value in result.dimensions.items()}
+        row["rubric_details"] = {
+            name: value.model_dump(mode="json") for name, value in result.dimensions.items()
+        }
+        row["hallucination_veto"] = result.veto_applied
+        row["hallucination_detail"] = result.hallucination.model_dump(mode="json")
+        row["evaluation_reasoning"] = result.reasoning
+        row["evaluation_suggestions"] = result.suggestions
+        updated += 1
+    if not errors:
+        payload["rubric_enrichment"] = {
+            "completed_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+            "evaluator": evaluator_type,
+            "model": model,
+            "records_updated": updated,
+            "all_ok_records_have_full_rubric": all(
+                row.get("status") != "ok"
+                or (
+                    set(row.get("rubric_details", {})) == REQUIRED_DIMENSIONS
+                    and row.get("hallucination_detail") is not None
+                )
+                for row in payload.get("records", [])
+            ),
+        }
+        temporary = path.with_suffix(path.suffix + ".tmp")
+        temporary.write_text(json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8")
+        temporary.replace(path)
+    return {"path": str(path), "updated": updated, "errors": errors}
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("checkpoint_dir", type=Path)
+    parser.add_argument("--workers", type=int, default=4)
+    parser.add_argument("--evaluator", default="kimi")
+    parser.add_argument("--model", default="kimi-k2.5")
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    paths = sorted(args.checkpoint_dir.glob("*.json"))
+    results = []
+    with ThreadPoolExecutor(max_workers=args.workers) as pool:
+        futures = {
+            pool.submit(enrich_case, path, args.evaluator, args.model, EVAL_DIR / "test_cases"): path
+            for path in paths
+        }
+        for index, future in enumerate(as_completed(futures), 1):
+            result = future.result()
+            results.append(result)
+            print(f"[{index}/{len(paths)}] {Path(result['path']).stem}: +{result['updated']} rubric records, errors={len(result['errors'])}")
+    report = {
+        "schema_version": "1.0",
+        "experiment": "7-3 rubric enrichment",
+        "generated_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "checkpoint_dir": str(args.checkpoint_dir),
+        "checkpoint_count": len(paths),
+        "records_updated": sum(row["updated"] for row in results),
+        "errors": [error for row in results for error in row["errors"]],
+        "complete": not any(row["errors"] for row in results),
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(report, indent=2, ensure_ascii=False), encoding="utf-8")
+    print(f"Wrote enrichment audit to {args.output}; complete={report['complete']}")
+    return 0 if report["complete"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter7/user-memory-system-evaluation/env.example

@@ -0,0 +1,8 @@
+OPENAI_API_KEY=your_openai_key
+KIMI_API_KEY=your_moonshot_key
+SILICONFLOW_API_KEY=your_siliconflow_key
+ARK_API_KEY=your_volcengine_ark_key
+MISTRAL_API_KEY=your_mistral_key
+# OpenRouter carries the BGE-M3/OpenAI/Qwen3 embeddings after the documented
+# 2026-07-31 substitutions (see default_config.yaml header and README.md).
+OPENROUTER_API_KEY=your_openrouter_key

chapter7/user-memory-system-evaluation/live_config.yaml

@@ -0,0 +1,64 @@
+# Known-working validation subset on the companion development account.
+# This never substitutes for default_config.yaml's full BGE/OpenAI/Doubao sweep.
+chat_models:
+  kimi:
+    model: kimi-k2.5
+    base_url: https://api.moonshot.cn/v1
+    api_key_env: KIMI_API_KEY
+    disable_thinking: true
+    temperature: 0.6
+    pricing:
+      currency: CNY
+      as_of_date: "2026-07-29"
+      source_url: https://platform.kimi.com/docs/pricing/chat-k25
+      input_per_million: 4.00
+      cached_input_per_million: 0.70
+      output_per_million: 21.00
+      source_note: Published Kimi K2.5 list price; input rate is cache-miss/uncached.
+  doubao:
+    model: doubao-seed-1-6-250615
+    base_url: https://ark.cn-beijing.volces.com/api/v3
+    api_key_env: ARK_API_KEY
+
+embeddings:
+  mistral:
+    model: mistral-embed
+    base_url: https://api.mistral.ai/v1
+    api_key_env: MISTRAL_API_KEY
+    pricing:
+      currency: USD
+      as_of_date: "2026-07-29"
+      source_url: https://mistral.ai/pricing/api/
+      input_per_million: 0.10
+      source_note: Published Mistral Embed list price.
+  codestral:
+    model: codestral-embed
+    base_url: https://api.mistral.ai/v1
+    api_key_env: MISTRAL_API_KEY
+
+rerankers:
+  none:
+    type: none
+  kimi-semantic:
+    type: llm
+    chat_model: kimi
+
+judge:
+  evaluator: kimi
+  model: kimi-k2.5
+
+experiment_7_4:
+  main_model: kimi
+  embedding: mistral
+  reranker: kimi-semantic
+  rounds_per_chunk: 6
+  overlap: 2
+
+experiment_7_11:
+  embeddings: [mistral, codestral]
+  rerankers: [none, kimi-semantic]
+  main_models: [kimi, doubao]
+  retrieval_judge_model: kimi
+  max_search_rounds: 3
+  rounds_per_chunk: 6
+  overlap: 2

chapter7/user-memory-system-evaluation/probe_backends.py

@@ -0,0 +1,128 @@
+#!/usr/bin/env python3
+"""Probe every backend required by a config without exposing credentials."""
+
+import argparse
+import json
+import os
+import time
+from pathlib import Path
+
+from experiment import (
+    ChatBackend,
+    Chunk,
+    EmbeddingBackend,
+    EndpointSpec,
+    ExperimentRunner,
+    execution_config_fingerprint,
+    load_config,
+    required_readiness_components,
+)
+
+
+def sanitized_error(exc: Exception, key_envs) -> str:
+    message = f"{type(exc).__name__}: {exc}"
+    for env_name in key_envs:
+        secret = os.getenv(env_name, "")
+        if secret:
+            message = message.replace(secret, "<redacted>")
+    return message[:1000]
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--config", type=Path, default=Path(__file__).with_name("default_config.yaml"))
+    parser.add_argument("--output", type=Path, required=True)
+    args = parser.parse_args()
+    config = load_config(args.config)
+    key_envs = {
+        data["api_key_env"]
+        for section in ("chat_models", "embeddings")
+        for data in config[section].values()
+    } | {
+        data["api_key_env"] for data in config["rerankers"].values() if data.get("api_key_env")
+    }
+    results = []
+
+    required_chat = {
+        config["experiment_7_4"]["main_model"],
+        config["experiment_7_11"]["retrieval_judge_model"],
+        *config["experiment_7_11"]["main_models"],
+        *[
+            data["chat_model"]
+            for name, data in config["rerankers"].items()
+            if name in config["experiment_7_11"]["rerankers"] and data.get("type") == "llm"
+        ],
+    }
+    for name in sorted(required_chat):
+        raw = config["chat_models"][name]
+        row = {"component": "chat", "name": name, "model": raw.get("model"), "key_env": raw.get("api_key_env")}
+        try:
+            spec = EndpointSpec.from_dict({"name": name, **raw})
+            turn = ChatBackend(spec).complete([{"role": "user", "content": "Reply exactly OK"}])
+            row.update(status="ok", latency_ms=turn.latency_ms, key_present=True)
+        except Exception as exc:
+            row.update(status="error", error=sanitized_error(exc, key_envs), key_present=bool(os.getenv(raw.get("api_key_env", ""))))
+        results.append(row)
+
+    required_embeddings = {
+        config["experiment_7_4"]["embedding"], *config["experiment_7_11"]["embeddings"]
+    }
+    for name in sorted(required_embeddings):
+        raw = config["embeddings"][name]
+        row = {"component": "embedding", "name": name, "model": raw.get("model"), "key_env": raw.get("api_key_env")}
+        try:
+            spec = EndpointSpec.from_dict({"name": name, **raw})
+            backend = EmbeddingBackend(spec)
+            vector = backend.embed(["user memory retrieval backend probe"])[0]
+            row.update(status="ok", dimensions=len(vector), latency_ms=backend.last_latency_ms, key_present=True)
+        except Exception as exc:
+            row.update(status="error", error=sanitized_error(exc, key_envs), key_present=bool(os.getenv(raw.get("api_key_env", ""))))
+        results.append(row)
+
+    # Reuse the production factory so this verifies the same reranker code path.
+    runner = object.__new__(ExperimentRunner)
+    runner.config = config
+    runner.endpoint_specs = {
+        name: EndpointSpec.from_dict({"name": name, **data}) for name, data in config["chat_models"].items()
+    }
+    chunks = [Chunk("a", "probe", "checking account number 123", 1, 1), Chunk("b", "probe", "weather", 2, 2)]
+    required_rerankers = {
+        config["experiment_7_4"]["reranker"], *config["experiment_7_11"]["rerankers"]
+    }
+    for name in sorted(required_rerankers):
+        row = {"component": "reranker", "name": name}
+        try:
+            backend = runner._reranker(name)
+            ranked = backend.rerank("checking account", chunks, 2)
+            row.update(status="ok", returned=len(ranked), latency_ms=backend.last_latency_ms)
+        except Exception as exc:
+            row.update(status="error", error=sanitized_error(exc, key_envs))
+        results.append(row)
+
+    payload = {
+        "schema_version": "2.0",
+        "experiment": "7-4/7-11 provider readiness",
+        "generated_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "config_file": str(args.config),
+        "execution_config_fingerprint": execution_config_fingerprint(config, "7-11"),
+        "required_components": [
+            {"component": component, "name": name}
+            for component, name in sorted(required_readiness_components(config))
+        ],
+        "credentials_redacted": True,
+        "probes": results,
+        "summary": {
+            "ok": sum(row["status"] == "ok" for row in results),
+            "error": sum(row["status"] == "error" for row in results),
+            "all_required_backends_ready": all(row["status"] == "ok" for row in results),
+        },
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8")
+    print(json.dumps(payload["summary"]))
+    print(f"Wrote sanitized backend readiness evidence to {args.output}")
+    return 0 if payload["summary"]["all_required_backends_ready"] else 2
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter7/user-memory-system-evaluation/probe_candidates.py

@@ -0,0 +1,181 @@
+#!/usr/bin/env python3
+"""Exploratory backend probes for Experiment 7-11 readiness (2026-07-31).
+
+Probes candidate substitutions with minimal real calls (1-line embed, 1-token
+chat, tiny rerank) and records sanitized, credential-free receipts. Secrets are
+read from the environment only; every recorded error string is scrubbed of any
+environment-held credential before being written.
+"""
+
+import json
+import os
+import time
+from pathlib import Path
+
+import requests
+from openai import OpenAI
+
+HERE = Path(__file__).resolve().parent
+OUT = HERE / "results" / "candidate_backend_probes_20260731.json"
+
+KEY_ENVS = [
+    "KIMI_API_KEY", "MOONSHOT_API_KEY", "ARK_API_KEY", "DASHSCOPE_API_KEY",
+    "SILICONFLOW_API_KEY", "MISTRAL_API_KEY", "GEMINI_API_KEY",
+    "OPENROUTER_API_KEY", "OPENAI_API_KEY",
+]
+
+
+def scrub(text: str) -> str:
+    for env in KEY_ENVS:
+        secret = os.getenv(env, "")
+        if secret:
+            text = text.replace(secret, "<redacted>")
+    return text[:1500]
+
+
+def embed_probe(name, base_url, key_env, model, **extra):
+    row = {"component": "embedding", "name": name, "model": model,
+           "base_url": base_url, "key_env": key_env,
+           "key_present": bool(os.getenv(key_env, ""))}
+    started = time.perf_counter()
+    try:
+        client = OpenAI(api_key=os.environ[key_env], base_url=base_url, timeout=60)
+        kwargs = {"model": model, "input": ["user memory retrieval backend probe"]}
+        kwargs.update(extra)
+        resp = client.embeddings.create(**kwargs)
+        row.update(status="ok", dimensions=len(resp.data[0].embedding),
+                   latency_ms=round((time.perf_counter() - started) * 1000, 1),
+                   usage=resp.usage.model_dump() if resp.usage else None)
+    except Exception as exc:  # noqa: BLE001 - receipts must capture any failure
+        row.update(status="error", latency_ms=round((time.perf_counter() - started) * 1000, 1),
+                   error=scrub(f"{type(exc).__name__}: {exc}"))
+    return row
+
+
+def chat_probe(name, base_url, key_env, model, max_tokens=1, **extra):
+    row = {"component": "chat", "name": name, "model": model,
+           "base_url": base_url, "key_env": key_env,
+           "key_present": bool(os.getenv(key_env, ""))}
+    started = time.perf_counter()
+    try:
+        client = OpenAI(api_key=os.environ[key_env], base_url=base_url, timeout=60)
+        kwargs = {"model": model,
+                  "messages": [{"role": "user", "content": "Reply exactly OK"}],
+                  "max_tokens": max_tokens}
+        kwargs.update(extra)
+        resp = client.chat.completions.create(**kwargs)
+        row.update(status="ok", content=(resp.choices[0].message.content or "")[:40],
+                   latency_ms=round((time.perf_counter() - started) * 1000, 1),
+                   usage=resp.usage.model_dump() if resp.usage else None)
+    except Exception as exc:  # noqa: BLE001
+        row.update(status="error", latency_ms=round((time.perf_counter() - started) * 1000, 1),
+                   error=scrub(f"{type(exc).__name__}: {exc}"))
+    return row
+
+
+def http_probe(name, method, url, key_env, payload=None):
+    row = {"component": "http", "name": name, "url": url, "key_env": key_env,
+           "key_present": bool(os.getenv(key_env, ""))}
+    started = time.perf_counter()
+    try:
+        headers = {"Authorization": f"Bearer {os.environ[key_env]}",
+                   "Content-Type": "application/json"}
+        resp = requests.request(method, url, headers=headers, json=payload, timeout=60)
+        row.update(status="ok" if resp.ok else "error", http_status=resp.status_code,
+                   latency_ms=round((time.perf_counter() - started) * 1000, 1),
+                   body=scrub(resp.text))
+    except Exception as exc:  # noqa: BLE001
+        row.update(status="error", latency_ms=round((time.perf_counter() - started) * 1000, 1),
+                   error=scrub(f"{type(exc).__name__}: {exc}"))
+    return row
+
+
+def main():
+    results = []
+
+    # --- SiliconFlow: reproduce and diagnose the 401 -------------------------
+    results.append(embed_probe(
+        "siliconflow-bge-m3", "https://api.siliconflow.cn/v1",
+        "SILICONFLOW_API_KEY", "BAAI/bge-m3"))
+    results.append(http_probe(
+        "siliconflow-rerank-v2-m3", "POST", "https://api.siliconflow.cn/v1/rerank",
+        "SILICONFLOW_API_KEY",
+        {"model": "BAAI/bge-reranker-v2-m3", "query": "checking account",
+         "documents": ["checking account number 123", "weather"], "top_n": 2,
+         "return_documents": False}))
+    # Account-level diagnosis: is the key itself dead or just the model/balance?
+    results.append(http_probe(
+        "siliconflow-user-info", "GET", "https://api.siliconflow.cn/v1/user/info",
+        "SILICONFLOW_API_KEY"))
+
+    # --- OpenAI direct: confirm quota state ----------------------------------
+    results.append(embed_probe(
+        "openai-text-embedding-3-small", "https://api.openai.com/v1",
+        "OPENAI_API_KEY", "text-embedding-3-small"))
+
+    # --- OpenRouter: OpenAI embedding pass-through + BGE-M3 availability -----
+    results.append(embed_probe(
+        "openrouter-openai-text-embedding-3-small", "https://openrouter.ai/api/v1",
+        "OPENROUTER_API_KEY", "openai/text-embedding-3-small"))
+    results.append(embed_probe(
+        "openrouter-baai-bge-m3", "https://openrouter.ai/api/v1",
+        "OPENROUTER_API_KEY", "BAAI/bge-m3"))
+
+    # --- ARK/Doubao: try public model-name embedding access ------------------
+    for model in ("doubao-embedding-large-text-250515",
+                  "doubao-embedding-large-text-240915",
+                  "doubao-embedding-text-240715"):
+        results.append(embed_probe(
+            f"ark-{model}", "https://ark.cn-beijing.volces.com/api/v3",
+            "ARK_API_KEY", model))
+
+    # --- DashScope (Alibaba): documented substitutes -------------------------
+    results.append(embed_probe(
+        "dashscope-text-embedding-v4", "https://dashscope.aliyuncs.com/compatible-mode/v1",
+        "DASHSCOPE_API_KEY", "text-embedding-v4"))
+    results.append(http_probe(
+        "dashscope-gte-rerank-v2", "POST",
+        "https://dashscope.aliyuncs.com/api/v1/services/rerank/text-rerank/text-rerank",
+        "DASHSCOPE_API_KEY",
+        {"model": "gte-rerank-v2",
+         "input": {"query": "checking account",
+                   "documents": ["checking account number 123", "weather"]},
+         "parameters": {"top_n": 2, "return_documents": False}}))
+
+    # --- Known-good controls --------------------------------------------------
+    results.append(embed_probe(
+        "mistral-embed", "https://api.mistral.ai/v1",
+        "MISTRAL_API_KEY", "mistral-embed"))
+    results.append(chat_probe(
+        "kimi-k2.5", "https://api.moonshot.cn/v1", "KIMI_API_KEY", "kimi-k2.5",
+        max_tokens=16, extra_body={"thinking": {"type": "disabled"}}, temperature=0.6))
+    results.append(chat_probe(
+        "doubao-seed-1-6-250615", "https://ark.cn-beijing.volces.com/api/v3",
+        "ARK_API_KEY", "doubao-seed-1-6-250615", max_tokens=16))
+
+    # --- Gemini embedding (last-resort fallback) ------------------------------
+    results.append(embed_probe(
+        "gemini-embedding-001", "https://generativelanguage.googleapis.com/v1beta/openai/",
+        "GEMINI_API_KEY", "gemini-embedding-001"))
+
+    payload = {
+        "schema_version": "1.0",
+        "purpose": "Experiment 7-11 readiness substitution probes",
+        "generated_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "credentials_redacted": True,
+        "probes": results,
+        "summary": {
+            "ok": sum(r["status"] == "ok" for r in results),
+            "error": sum(r["status"] == "error" for r in results),
+        },
+    }
+    OUT.parent.mkdir(parents=True, exist_ok=True)
+    OUT.write_text(json.dumps(payload, indent=2, ensure_ascii=False), encoding="utf-8")
+    for row in results:
+        print(f"{row['status']:5s} {row['name']}")
+    print(json.dumps(payload["summary"]))
+    print(f"Wrote {OUT}")
+
+
+if __name__ == "__main__":
+    main()

chapter7/user-memory-system-evaluation/rebuild_report.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python3
+"""Rebuild aggregate/report metadata from saved live trajectory records.
+
+This performs no model calls and never invents records. It is useful when report
+logic changes (for example, the stricter core-dimension success gate) while the
+underlying expensive API answers and judge dimension scores remain valid.
+"""
+
+import argparse
+import hashlib
+import json
+import time
+from pathlib import Path
+
+from experiment import RunRecord, load_config, reprice_legacy_64_records, save_report
+
+
+CORE_SUCCESS_DIMENSIONS = ("precision", "recall", "reasoning")
+ACCOUNTING_FIELDS = {
+    "cost_usd",
+    "cost_by_currency",
+    "unpriced_tokens",
+    "cached_input_tokens",
+    "unpriced_requests",
+    "cost_accounting",
+}
+
+
+def canonical_non_accounting_hash(records: list[RunRecord]) -> str:
+    payload = [
+        {
+            key: value
+            for key, value in vars(record).items()
+            if key not in ACCOUNTING_FIELDS
+        }
+        for record in records
+    ]
+    encoded = json.dumps(
+        payload,
+        sort_keys=True,
+        ensure_ascii=False,
+        separators=(",", ":"),
+    ).encode("utf-8")
+    return hashlib.sha256(encoded).hexdigest()
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("source", type=Path)
+    parser.add_argument("output", type=Path)
+    parser.add_argument("--config", type=Path, default=Path(__file__).with_name("live_config.yaml"))
+    parser.add_argument(
+        "--reprice-legacy-7-4",
+        action="store_true",
+        help=(
+            "Cover legacy Kimi usage with dated native-CNY list prices. Legacy cached-token counts "
+            "were not saved, so all unpriced Kimi input uses the uncached rate."
+        ),
+    )
+    args = parser.parse_args()
+
+    source = json.loads(args.source.read_text(encoding="utf-8"))
+    records = [RunRecord(**row) for row in source.get("records", [])]
+    non_accounting_hash_before = canonical_non_accounting_hash(records)
+    config = load_config(args.config)
+    repricing = None
+    if args.reprice_legacy_7_4:
+        if source.get("experiment") != "7-4":
+            parser.error("--reprice-legacy-7-4 requires an Experiment 7-4 source")
+        repricing = reprice_legacy_64_records(
+            records,
+            config,
+            source_generated_at_utc=source.get("generated_at_utc"),
+        )
+    for record in records:
+        if record.status != "ok":
+            record.success = False
+            continue
+        record.success = (
+            not record.hallucination_veto
+            and all(record.rubric_dimensions.get(name, 0) >= 3 for name in CORE_SUCCESS_DIMENSIONS)
+        )
+    non_accounting_hash_after = canonical_non_accounting_hash(records)
+    if non_accounting_hash_before != non_accounting_hash_after:
+        raise RuntimeError(
+            "canonical non-accounting record hash changed; refusing to write a derived report"
+        )
+    save_report(args.output, source["experiment"], records, config)
+    rebuilt = json.loads(args.output.read_text(encoding="utf-8"))
+    rebuilt["evidence_lineage"] = {
+        "source_file": str(args.source),
+        "source_api_generated_at_utc": source.get("generated_at_utc"),
+        "report_rebuilt_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
+        "transformation": (
+            "No API records were added or removed. Aggregates/run-scope metadata were rebuilt and "
+            "success was recomputed from saved rubric dimensions using the documented >=3 core gate. "
+            + (
+                "Legacy unpriced Kimi usage was covered using dated native-CNY list prices; all legacy "
+                "input was conservatively treated as uncached because cached-token counts were not saved."
+                if repricing else "No cost repricing was requested."
+            )
+        ),
+        "repricing": repricing,
+        "trajectory_record_count_preserved": len(records),
+        "canonical_non_accounting_sha256_before": non_accounting_hash_before,
+        "canonical_non_accounting_sha256_after": non_accounting_hash_after,
+        "canonical_non_accounting_records_preserved": True,
+        "canonical_hash_excluded_fields": sorted(ACCOUNTING_FIELDS),
+    }
+    args.output.write_text(json.dumps(rebuilt, indent=2, ensure_ascii=False), encoding="utf-8")
+    print(f"Rebuilt {len(records)} saved trajectories into {args.output}; no model calls made")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter7/user-memory-system-evaluation/requirements.txt

@@ -0,0 +1,8 @@
+openai>=1.0
+pydantic>=2.0
+python-dotenv>=1.0
+pyyaml>=6.0
+requests>=2.31
+rich>=13.0
+tenacity>=8.0
+pytest>=8.0

chapter7/user-memory-system-evaluation/run_full.py

@@ -0,0 +1,312 @@
+#!/usr/bin/env python3
+"""Restart-safe bounded-parallel runner for the full 60-case experiments."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import subprocess
+import sys
+import time
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Set, Tuple
+
+from experiment import (
+    EVAL_DIR,
+    HERE,
+    RunRecord,
+    UserMemoryEvaluationFramework,
+    execution_config_fingerprint,
+    load_config,
+    reprice_legacy_64_records,
+    save_report,
+    validate_readiness,
+)
+
+
+REQUIRED_RUBRIC_DIMENSIONS = {"precision", "recall", "reasoning", "proactivity"}
+
+
+Cell = Tuple[str, ...]
+
+
+def valid_checkpoint(
+    path: Path,
+    test_id: str,
+    experiment: str,
+    expected_records: int,
+    required_ok_cells: Optional[Set[Cell]] = None,
+    expected_cells: Optional[Set[Cell]] = None,
+    expected_config_fingerprint: Optional[str] = None,
+) -> bool:
+    try:
+        data = json.loads(path.read_text(encoding="utf-8"))
+    except (OSError, json.JSONDecodeError):
+        return False
+    scope = data.get("run_scope", {})
+    records = data.get("records", [])
+    if not (
+        data.get("experiment") == experiment
+        and scope.get("requested_test_ids") == [test_id]
+        and len(records) == expected_records
+    ):
+        return False
+    if expected_config_fingerprint is not None:
+        try:
+            observed_fingerprint = execution_config_fingerprint(
+                data["configuration"], experiment
+            )
+        except (KeyError, TypeError, ValueError):
+            return False
+        if observed_fingerprint != expected_config_fingerprint:
+            return False
+    observed_cells: List[Cell] = []
+    for row in records:
+        if row.get("test_id") != test_id or row.get("experiment") != experiment:
+            return False
+        if row.get("status") == "ok":
+            if set(row.get("rubric_details", {})) != REQUIRED_RUBRIC_DIMENSIONS:
+                return False
+            if row.get("hallucination_detail") is None:
+                return False
+        elif row.get("status") == "error":
+            identity = (row.get("embedding"), row.get("reranker"), row.get("main_model"))
+            if (
+                experiment == "7-4"
+                or not row.get("error")
+                or required_ok_cells is None
+                or identity in required_ok_cells
+            ):
+                return False
+        else:
+            return False
+        if experiment == "7-11":
+            observed_cells.append((row.get("embedding"), row.get("reranker"), row.get("main_model")))
+        else:
+            observed_cells.append((row.get("system"),))
+    if expected_cells is not None and (
+        len(observed_cells) != len(set(observed_cells)) or set(observed_cells) != expected_cells
+    ):
+        return False
+    return True
+
+
+def required_611_cells(config: Dict[str, Any], readiness: Optional[Dict[str, Any]]) -> Set[Cell]:
+    """Return matrix cells whose backends passed preflight and must complete live."""
+    matrix = config["experiment_7_11"]
+    blocked = {
+        (row["component"], row["name"])
+        for row in (readiness or {}).get("probes", [])
+        if row.get("status") == "error"
+    }
+    return {
+        (embedding, reranker, main_model)
+        for embedding in matrix["embeddings"]
+        for reranker in matrix["rerankers"]
+        for main_model in matrix["main_models"]
+        if ("embedding", embedding) not in blocked
+        and ("reranker", reranker) not in blocked
+        and ("chat", main_model) not in blocked
+    }
+
+
+def run_case(
+    experiment: str,
+    config_path: Path,
+    test_id: str,
+    output: Path,
+    readiness: Path | None = None,
+) -> Dict[str, Any]:
+    command = [
+        sys.executable,
+        str(HERE / "experiment.py"),
+        experiment,
+        "--config",
+        str(config_path),
+        "--test-id",
+        test_id,
+        "--output",
+        str(output),
+    ]
+    if readiness:
+        command.extend(["--readiness", str(readiness)])
+    started = time.perf_counter()
+    process = subprocess.run(command, cwd=HERE, capture_output=True, text=True)
+    return {
+        "test_id": test_id,
+        "returncode": process.returncode,
+        "elapsed_seconds": time.perf_counter() - started,
+        "stdout": process.stdout[-2000:],
+        "stderr": process.stderr[-4000:],
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("experiment", choices=["7-4", "7-11"])
+    parser.add_argument("--config", type=Path, default=HERE / "default_config.yaml")
+    parser.add_argument("--workers", type=int, default=4)
+    parser.add_argument("--checkpoint-dir", type=Path)
+    parser.add_argument("--output", type=Path, required=True)
+    parser.add_argument("--readiness", type=Path)
+    args = parser.parse_args()
+    if args.workers < 1:
+        parser.error("--workers must be at least 1")
+
+    config_path = args.config.resolve()
+    config = load_config(config_path)
+    readiness_data = (
+        json.loads(args.readiness.resolve().read_text(encoding="utf-8"))
+        if args.readiness else None
+    )
+    if args.experiment == "7-11":
+        if readiness_data is None:
+            parser.error("exact Experiment 7-11 requires --readiness from probe_backends.py")
+        readiness_errors = validate_readiness(config, readiness_data)
+        if readiness_errors:
+            parser.error("invalid readiness evidence: " + "; ".join(readiness_errors))
+        config["execution_readiness"] = {
+            "source_file": str(args.readiness.resolve()),
+            "generated_at_utc": readiness_data.get("generated_at_utc"),
+            "execution_config_fingerprint": readiness_data.get("execution_config_fingerprint"),
+            "all_required_backends_ready": readiness_data.get("summary", {}).get("all_required_backends_ready"),
+            "validated": True,
+        }
+        if not readiness_data.get("summary", {}).get("all_required_backends_ready"):
+            parser.error(
+                "exact Experiment 7-11 campaign is blocked: every required real backend "
+                "must pass probe_backends.py before launch"
+            )
+    framework = UserMemoryEvaluationFramework(str(EVAL_DIR / "test_cases"))
+    test_ids = [case.test_id for case in framework.list_test_cases()]
+    if len(test_ids) != 60:
+        parser.error(f"full run requires exactly 60 loaded cases, found {len(test_ids)}")
+
+    if args.experiment == "7-4":
+        expected_records = 3
+        required_ok_cells = None
+        all_matrix_cells = {
+            ("advanced_json_cards",),
+            ("rag",),
+            ("hybrid",),
+        }
+    else:
+        matrix = config["experiment_7_11"]
+        shape = (len(matrix["embeddings"]), len(matrix["rerankers"]), len(matrix["main_models"]))
+        if shape != (4, 3, 2):
+            parser.error(f"exact Experiment 7-11 requires a 4x3x2 matrix, found {shape}")
+        expected_records = len(matrix["embeddings"]) * len(matrix["rerankers"]) * len(matrix["main_models"])
+        required_ok_cells = required_611_cells(config, readiness_data)
+        all_matrix_cells = {
+            (embedding, reranker, main_model)
+            for embedding in matrix["embeddings"]
+            for reranker in matrix["rerankers"]
+            for main_model in matrix["main_models"]
+        }
+    checkpoint_dir = args.checkpoint_dir or (
+        HERE / "results" / "checkpoints" / args.experiment.replace("-", "_") / config_path.stem
+    )
+    checkpoint_dir.mkdir(parents=True, exist_ok=True)
+    expected_fingerprint = execution_config_fingerprint(config, args.experiment)
+
+    pending = []
+    for test_id in test_ids:
+        path = checkpoint_dir / f"{test_id}.json"
+        if not valid_checkpoint(
+            path,
+            test_id,
+            args.experiment,
+            expected_records,
+            required_ok_cells,
+            all_matrix_cells,
+            expected_fingerprint,
+        ):
+            pending.append((test_id, path))
+    print(f"Full {args.experiment}: {60 - len(pending)}/60 checkpoints reusable; {len(pending)} pending")
+
+    failures: List[Dict[str, Any]] = []
+    with ThreadPoolExecutor(max_workers=args.workers) as pool:
+        futures = {
+            pool.submit(
+                run_case,
+                args.experiment,
+                config_path,
+                test_id,
+                path,
+                args.readiness.resolve() if args.readiness else None,
+            ): test_id
+            for test_id, path in pending
+        }
+        done = 60 - len(pending)
+        for future in as_completed(futures):
+            result = future.result()
+            done += 1
+            if result["returncode"]:
+                failures.append(result)
+                print(f"[{done}/60] ERROR {result['test_id']} ({result['elapsed_seconds']:.1f}s)")
+            else:
+                print(f"[{done}/60] OK {result['test_id']} ({result['elapsed_seconds']:.1f}s)")
+
+    records: List[RunRecord] = []
+    valid_case_ids = []
+    for test_id in test_ids:
+        path = checkpoint_dir / f"{test_id}.json"
+        if not valid_checkpoint(
+            path,
+            test_id,
+            args.experiment,
+            expected_records,
+            required_ok_cells,
+            all_matrix_cells,
+            expected_fingerprint,
+        ):
+            continue
+        data = json.loads(path.read_text(encoding="utf-8"))
+        records.extend(RunRecord(**row) for row in data["records"])
+        valid_case_ids.append(test_id)
+
+    repricing = None
+    if args.experiment == "7-4":
+        repricing = reprice_legacy_64_records(records, config)
+    save_report(args.output, args.experiment, records, config)
+    merged = json.loads(args.output.read_text(encoding="utf-8"))
+    merged["full_run_orchestration"] = {
+        "workers": args.workers,
+        "checkpoint_dir": str(checkpoint_dir),
+        "expected_case_count": 60,
+        "expected_records_per_case": expected_records,
+        "valid_checkpoint_case_count": len(valid_case_ids),
+        "missing_case_ids": sorted(set(test_ids) - set(valid_case_ids)),
+        "subprocess_failures": failures,
+        "execution_config_fingerprint": expected_fingerprint,
+        "legacy_7_4_repricing": repricing,
+    }
+    # A complete experiment requires exact case/cell coverage, real successful
+    # trajectories, explicit readiness (7-11), and zero unpriced usage.
+    complete = bool(merged["completion"]["evidence_complete"]) and not failures
+    if failures:
+        merged["completion"]["evidence_complete"] = False
+        merged["completion"]["trajectory_matrix_complete"] = False
+        merged["completion"]["status"] = "incomplete"
+        merged["completion"]["blockers"].append({
+            "code": "subprocess_failures",
+            "message": f"{len(failures)} case subprocesses failed",
+        })
+        merged["status"] = "incomplete"
+    merged["run_scope"]["full_60_case_suite_completed"] = complete
+    merged["run_scope"]["validation_scope"] = (
+        "full" if complete else "incomplete-full-suite"
+    )
+    args.output.write_text(json.dumps(merged, indent=2, ensure_ascii=False), encoding="utf-8")
+    print(
+        f"Merged {len(records)} records from {len(valid_case_ids)}/60 cases into {args.output}; "
+        f"full completion={complete}"
+    )
+    if complete:
+        return 0
+    return 2 if merged["completion"]["status"] == "blocked" else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter7/user-memory-system-evaluation/supplement_73_missing.py

@@ -0,0 +1,115 @@
+#!/usr/bin/env python3
+"""Re-judge only incomplete saved 7-3 rubric records without mutating 7-4.
+
+The source campaign remains immutable.  Each supplemental judgment records the
+source record identity and answer hash so the full 7-3 validator can join it
+without confusing it with a newly executed memory-system trajectory.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+EVAL_DIR = HERE.parents[1] / "chapter3" / "user-memory-evaluation"
+sys.path.insert(0, str(EVAL_DIR))
+
+from evaluator import LLMEvaluator  # noqa: E402
+from framework import UserMemoryEvaluationFramework  # noqa: E402
+
+
+REQUIRED_DIMENSIONS = {"precision", "recall", "reasoning", "proactivity"}
+
+
+def answer_hash(answer: str) -> str:
+    return hashlib.sha256(answer.encode("utf-8")).hexdigest()
+
+
+def complete_rubric(row: dict) -> bool:
+    dimensions = row.get("rubric_details") or {}
+    hallucination = row.get("hallucination_detail")
+    return (
+        set(dimensions) == REQUIRED_DIMENSIONS
+        and isinstance(hallucination, dict)
+        and "detected" in hallucination
+        and all(
+            isinstance(detail, dict)
+            and bool(detail.get("reasoning"))
+            and bool(detail.get("evidence") or detail.get("boundary_case"))
+            for detail in dimensions.values()
+        )
+    )
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser()
+    parser.add_argument(
+        "--source",
+        type=Path,
+        default=HERE / "results" / "full_7_4_60_cases_costed.json",
+    )
+    parser.add_argument(
+        "--output",
+        type=Path,
+        default=HERE / "results" / "full_7_3_missing_rubric_supplement.json",
+    )
+    parser.add_argument("--evaluator", default="kimi", choices=["kimi", "openai"])
+    parser.add_argument("--model", default="kimi-k2.5")
+    args = parser.parse_args()
+
+    source = json.loads(args.source.read_text(encoding="utf-8"))
+    missing = [row for row in source.get("records", []) if not complete_rubric(row)]
+    framework = UserMemoryEvaluationFramework(str(EVAL_DIR / "test_cases"))
+    judge = LLMEvaluator(args.evaluator, model=args.model)
+    supplements = []
+    for row in missing:
+        test_case = framework.get_test_case(row["test_id"])
+        result = judge.evaluate(test_case, row["answer"])
+        supplements.append({
+            "test_id": row["test_id"],
+            "system": row["system"],
+            "layer": row["layer"],
+            "answer_sha256": answer_hash(row["answer"]),
+            "provider": args.evaluator,
+            "model": args.model,
+            "evaluation": result.model_dump(mode="json"),
+        })
+        print(f"Re-judged {row['test_id']} / {row['system']}")
+
+    complete = all(
+        set(item["evaluation"].get("dimensions", {})) == REQUIRED_DIMENSIONS
+        and item["evaluation"].get("hallucination") is not None
+        and all(
+            detail.get("reasoning") and (detail.get("evidence") or detail.get("boundary_case"))
+            for detail in item["evaluation"]["dimensions"].values()
+        )
+        for item in supplements
+    )
+    report = {
+        "schema_version": "1.0",
+        "experiment": "7-3",
+        "purpose": "supplement incomplete rubric evidence only; source trajectories are unchanged",
+        "generated_at_utc": datetime.now(timezone.utc).isoformat(),
+        "source_file": str(args.source),
+        "missing_records_detected": len(missing),
+        "supplements": supplements,
+        "status": "complete" if complete else "incomplete",
+    }
+    args.output.parent.mkdir(parents=True, exist_ok=True)
+    args.output.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
+    print(json.dumps({
+        "status": report["status"],
+        "supplements": len(supplements),
+        "output": str(args.output),
+    }))
+    return 0 if complete else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter7/user-memory-system-evaluation/test_experiment.py

@@ -0,0 +1,367 @@
+"""Offline tests for the experiment harness; live API evidence is stored separately."""
+
+import json
+from pathlib import Path
+
+from experiment import (
+    AgentResult,
+    CardBuilder,
+    ChatTurn,
+    Chunk,
+    MemoryAgent,
+    LLMReranker,
+    NoReranker,
+    ToolCall,
+    TokenPricing,
+    Usage,
+    VectorMemoryIndex,
+    aggregate,
+    completion_assessment,
+    conversation_chunks,
+    interaction_analysis,
+    pricing_coverage,
+    retrieval_metrics,
+    select_core_cards,
+)
+
+
+class FakeEmbedder:
+    def __init__(self):
+        self.last_usage = Usage()
+        self.last_latency_ms = 0.1
+
+    def embed(self, texts):
+        self.last_usage = Usage(input_tokens=len(texts), cost_usd=0.001 * len(texts))
+        vectors = []
+        for text in texts:
+            lower = text.lower()
+            vectors.append([float("checking" in lower), float("medical" in lower), 1.0])
+        return vectors
+
+
+class FakeToolChat:
+    def __init__(self):
+        self.calls = 0
+
+    def complete(self, messages, tools=None, tool_choice=None):
+        self.calls += 1
+        if self.calls == 1:
+            return ChatTurn(
+                "",
+                [ToolCall("call-1", "search_memory", {"query": "checking account"})],
+                Usage(input_tokens=10, output_tokens=2, cost_usd=0.01),
+                4.0,
+                {
+                    "role": "assistant",
+                    "content": "",
+                    "tool_calls": [{"id": "call-1", "type": "function", "function": {"name": "search_memory", "arguments": '{"query":"checking account"}'}}],
+                },
+            )
+        return ChatTurn(
+            "The checking account is 4429853327.", [], Usage(input_tokens=20, output_tokens=8, cost_usd=0.02), 5.0,
+            {"role": "assistant", "content": "The checking account is 4429853327."},
+        )
+
+
+class FakeMultiSearchChat(FakeToolChat):
+    def complete(self, messages, tools=None, tool_choice=None):
+        self.calls += 1
+        if self.calls <= 2:
+            call_id = f"call-{self.calls}"
+            query = "checking account" if self.calls == 1 else "routing number"
+            return ChatTurn(
+                "",
+                [ToolCall(call_id, "search_memory", {"query": query})],
+                Usage(input_tokens=5, output_tokens=2),
+                1.0,
+                {
+                    "role": "assistant",
+                    "content": "",
+                    "tool_calls": [{
+                        "id": call_id,
+                        "type": "function",
+                        "function": {"name": "search_memory", "arguments": json.dumps({"query": query})},
+                    }],
+                },
+            )
+        return ChatTurn("complete", [], Usage(input_tokens=5, output_tokens=1), 1.0,
+                        {"role": "assistant", "content": "complete"})
+
+
+def load_case(test_id="layer1_01_bank_account"):
+    import sys
+
+    eval_dir = Path(__file__).resolve().parents[2] / "chapter3" / "user-memory-evaluation"
+    if str(eval_dir) not in sys.path:
+        sys.path.insert(0, str(eval_dir))
+    from framework import UserMemoryEvaluationFramework
+
+    return UserMemoryEvaluationFramework(str(eval_dir / "test_cases")).get_test_case(test_id)
+
+
+def test_chunking_is_stable_and_preserves_source():
+    case = load_case()
+    first = conversation_chunks(case, rounds_per_chunk=8, overlap=2)
+    second = conversation_chunks(case, rounds_per_chunk=8, overlap=2)
+    assert [c.chunk_id for c in first] == [c.chunk_id for c in second]
+    assert len(first) > 1
+    assert all(c.conversation_id == "bank_setup_001" for c in first)
+    assert "4429853327" in "\n".join(c.text for c in first)
+
+
+def test_rag_agent_executes_real_tool_loop_shape_and_tracks_metrics():
+    chunks = [
+        Chunk("a", "c1", "checking account 4429853327", 1, 1),
+        Chunk("b", "c1", "medical appointment", 2, 2),
+    ]
+    index = VectorMemoryIndex(chunks, FakeEmbedder())
+    result = MemoryAgent(FakeToolChat()).rag("What is my checking account?", index, NoReranker())
+    assert result.answer.endswith("4429853327.")
+    assert result.steps == 2
+    assert result.tool_calls == 1
+    assert result.retrieved_chunks[0].chunk_id == "a"
+    assert result.latency_ms > 9
+    assert result.usage.cost_usd >= 0.031
+
+
+def test_611_agent_can_make_followup_searches_and_exposes_tool_efficiency():
+    chunks = [Chunk("a", "c", "checking account routing number", 1, 1)]
+    index = VectorMemoryIndex(chunks, FakeEmbedder())
+    result = MemoryAgent(FakeMultiSearchChat()).rag(
+        "account?", index, NoReranker(), allow_followup_searches=True, max_search_rounds=3
+    )
+    assert result.answer == "complete"
+    assert result.steps == 3
+    assert result.tool_calls == 2
+    assert [event["event"] for event in result.trace] == ["search_memory", "search_memory", "answer"]
+
+
+def test_retrieval_metrics_use_source_selected_gold():
+    retrieved = [Chunk("wrong", "c", "", 1, 1), Chunk("gold", "c", "", 2, 2)]
+    hit, recall, mrr = retrieval_metrics(retrieved, ["gold", "also-gold"])
+    assert hit == 1.0
+    assert recall == 0.5
+    assert mrr == 0.5
+
+
+def test_hybrid_resident_context_contains_only_explicit_core_cards():
+    cards = [
+        {"card_key": "identity", "memory_tier": "core"},
+        {"card_key": "old_call_detail", "memory_tier": "supporting"},
+    ]
+    assert [card["card_key"] for card in select_core_cards(cards, "t")] == ["identity"]
+
+
+def test_hybrid_requires_at_least_one_core_card():
+    import pytest
+
+    with pytest.raises(RuntimeError, match="no core-tier cards"):
+        select_core_cards([{"memory_tier": "supporting"}], "case-x")
+
+
+class FakeCardChat:
+    def __init__(self):
+        self.calls = 0
+
+    def complete(self, messages, tools=None, tool_choice=None, json_object=False):
+        self.calls += 1
+        if self.calls == 1:
+            content = '{"cards": [{"category": "identity"}]}'
+        elif self.calls == 2:
+            content = "{malformed"
+        else:
+            content = json.dumps({
+                "cards": [{
+                    "category": "identity",
+                    "card_key": "user",
+                    "backstory": "account setup",
+                    "date_created": "2025-01-01",
+                    "person": "user",
+                    "relationship": "self",
+                    "facts": {"name": "Alex"},
+                    "source_conversation_ids": ["c1"],
+                    "status": "current",
+                    "memory_tier": "core",
+                }]
+            })
+        return ChatTurn(
+            content, [], Usage(input_tokens=10, output_tokens=5), 2.0,
+            {"role": "assistant", "content": content},
+        )
+
+
+def test_card_builder_retries_parse_and_schema_failures_without_weakening_schema():
+    chat = FakeCardChat()
+    cards, usage, latency = CardBuilder(chat).build(load_case())
+    assert chat.calls == 3
+    assert cards[0]["memory_tier"] == "core"
+    assert cards[0]["source_conversation_ids"] == ["c1"]
+    assert usage.input_tokens == 30
+    assert usage.output_tokens == 15
+    assert latency == 6.0
+
+
+class FakeRerankerChat:
+    def __init__(self):
+        self.messages = []
+        self.calls = 0
+
+    def complete(self, messages, tools=None, tool_choice=None, json_object=False):
+        self.calls += 1
+        self.messages.append(messages)
+        content = '{"ranking": []}' if self.calls == 1 else '{"ranking": [{"index": 1, "score": 0.9}]}'
+        return ChatTurn(
+            content, [], Usage(input_tokens=4, output_tokens=2), 1.5,
+            {"role": "assistant", "content": content},
+        )
+
+
+def test_llm_reranker_retry_corrects_an_empty_semantic_response():
+    chat = FakeRerankerChat()
+    reranker = LLMReranker("semantic", chat)
+    documents = [Chunk("a", "c", "first", 1, 1), Chunk("b", "c", "second", 2, 2)]
+    ranked = reranker.rerank("second", documents, 1)
+    assert [row[0].chunk_id for row in ranked] == ["b"]
+    assert chat.calls == 2
+    assert "previous response was invalid" in chat.messages[1][-1]["content"]
+    assert reranker.last_usage.input_tokens == 8
+    assert reranker.last_latency_ms == 3.0
+
+
+def record(**overrides):
+    from experiment import RunRecord
+
+    data = dict(
+        experiment="7-11", test_id="t", layer="layer1", system="rag", embedding="e1",
+        reranker="none", main_model="m1", success=True, reward=0.8, steps=2,
+        tool_calls=1, latency_ms=100, cost_usd=0.01, input_tokens=10, output_tokens=2,
+        unpriced_tokens=0, retrieval_hit_at_5=1.0, retrieval_recall_at_5=0.5,
+        retrieval_mrr=1.0, fixed_query_hit_at_5=1.0, fixed_query_recall_at_5=0.5,
+        fixed_query_mrr=1.0,
+    )
+    data.update(overrides)
+    return RunRecord(**data)
+
+
+def test_aggregation_and_interaction_report_conditional_reranker_value():
+    rows = [
+        record(test_id="a"),
+        record(test_id="b", success=False, reward=0.3),
+        record(test_id="a", reranker="bge", success=True, reward=1.0, retrieval_recall_at_5=1.0, fixed_query_recall_at_5=1.0, latency_ms=130),
+        record(test_id="b", reranker="bge", success=True, reward=0.9, retrieval_recall_at_5=1.0, fixed_query_recall_at_5=1.0, latency_ms=130),
+    ]
+    summary = aggregate(rows, ["reranker"])
+    assert {row["reranker"] for row in summary} == {"none", "bge"}
+    analysis = interaction_analysis(rows)
+    assert analysis["analysis_scope"]["selection_conclusions_allowed"] is False
+    assert analysis["analysis_scope"]["scope_status"] == "partial_descriptive_only"
+    delta = analysis["reranker_value_by_embedding_and_main_model"][0]
+    assert delta["success_rate_delta"] == 0.5
+    assert delta["fixed_query_recall_at_5_delta"] == 0.5
+    assert delta["latency_ms_delta"] == 30
+
+
+def test_native_currency_pricing_tracks_cached_and_uncached_without_fx():
+    pricing = TokenPricing.from_dict({
+        "currency": "CNY",
+        "as_of_date": "2026-07-29",
+        "source_url": "https://provider.example/pricing",
+        "input_per_million": 4.0,
+        "cached_input_per_million": 0.7,
+        "output_per_million": 21.0,
+    })
+    usage = pricing.price(1_000_000, 1_000_000, cached_input_tokens=250_000)
+    assert usage.cost_by_currency == {"CNY": 24.175}
+    assert usage.cost_usd == 0
+    assert usage.cached_input_tokens == 250_000
+    assert usage.unpriced_tokens == 0
+
+
+def test_pricing_requires_dated_source_and_three_letter_currency():
+    import pytest
+
+    base = {
+        "currency": "USD",
+        "as_of_date": "2026-07-29",
+        "source_url": "https://provider.example/pricing",
+        "input_per_million": 1.0,
+    }
+    for override in (
+        {"currency": "dollars"},
+        {"as_of_date": "today"},
+        {"source_url": "provider.example/pricing"},
+    ):
+        with pytest.raises(ValueError):
+            TokenPricing.from_dict(base | override)
+
+
+def exact_711_config(readiness=True):
+    return {
+        "experiment_7_11": {
+            "embeddings": ["e1", "e2", "e3", "e4"],
+            "rerankers": ["none", "r1", "r2"],
+            "main_models": ["m1", "m2"],
+        },
+        "execution_readiness": {"all_required_backends_ready": readiness},
+    }
+
+
+def exact_711_records():
+    return [
+        record(
+            test_id=f"case-{case:02d}",
+            embedding=embedding,
+            reranker=reranker,
+            main_model=model,
+            cost_usd=0,
+            cost_by_currency={"USD": 0.001},
+        )
+        for case in range(60)
+        for embedding in ("e1", "e2", "e3", "e4")
+        for reranker in ("none", "r1", "r2")
+        for model in ("m1", "m2")
+    ]
+
+
+def assess_611(rows, readiness=True):
+    return completion_assessment(
+        "7-11", rows, exact_711_config(readiness), pricing_coverage(rows)
+    )
+
+
+def test_exact_711_gate_requires_all_1440_real_priced_successes_and_readiness():
+    rows = exact_711_records()
+    complete = assess_611(rows)
+    assert complete["evidence_complete"] is True
+    assert complete["expected_full_trajectory_count"] == 1440
+    assert interaction_analysis(rows, complete)["analysis_scope"]["selection_conclusions_allowed"] is True
+
+    assert assess_611(rows[:-1])["evidence_complete"] is False
+
+    failed = list(rows)
+    failed[0] = record(**({
+        **vars(rows[0]), "status": "error", "success": False, "error": "provider failed",
+    }))
+    assert assess_611(failed)["evidence_complete"] is False
+
+    unpriced = list(rows)
+    unpriced[0] = record(**({**vars(rows[0]), "unpriced_tokens": 1}))
+    assert assess_611(unpriced)["cost_accounting_complete"] is False
+
+    non_api = list(rows)
+    non_api[0] = record(**({**vars(rows[0]), "evidence_mode": "mock"}))
+    assert assess_611(non_api)["real_api_evidence_only"] is False
+    assert assess_611(rows, readiness=False)["backend_readiness_complete"] is False
+
+
+def test_pricing_coverage_recovers_schema1_fixed_query_usd():
+    row = record(
+        cost_usd=0.01,
+        fixed_query_retrieval_cost_usd=0.002,
+        fixed_query_retrieval_cost_by_currency={},
+        fixed_query_unpriced_tokens=3,
+    )
+    coverage = pricing_coverage([row])
+    assert coverage["total_cost_by_currency"] == {"USD": 0.012}
+    assert coverage["observed_token_count"] == 15
+    assert coverage["legacy_fixed_query_unpriced_token_lower_bound"] == 3

chapter7/user-memory-system-evaluation/test_run_full.py

@@ -0,0 +1,91 @@
+"""Offline checkpoint-validity tests for the resumable full runner."""
+
+import json
+
+from run_full import required_611_cells, valid_checkpoint
+
+
+def record(experiment="7-4", status="ok", system="advanced_json_cards"):
+    row = {
+        "experiment": experiment,
+        "test_id": "case-1",
+        "status": status,
+        "error": None,
+        "rubric_details": {
+            name: {"score": 4}
+            for name in ("precision", "recall", "reasoning", "proactivity")
+        },
+        "hallucination_detail": {"detected": False},
+        "embedding": "e",
+        "reranker": "none",
+        "main_model": "m",
+        "system": system,
+    }
+    if status == "error":
+        row["error"] = "provider unavailable"
+        row["rubric_details"] = {}
+        row["hallucination_detail"] = None
+    return row
+
+
+def write_checkpoint(path, experiment, records):
+    path.write_text(json.dumps({
+        "experiment": experiment,
+        "run_scope": {"requested_test_ids": ["case-1"]},
+        "records": records,
+    }))
+
+
+def test_64_checkpoint_requires_successful_full_rubric_rows(tmp_path):
+    path = tmp_path / "case.json"
+    rows = [record(system=name) for name in ("advanced_json_cards", "rag", "hybrid")]
+    write_checkpoint(path, "7-4", rows)
+    expected = {("advanced_json_cards",), ("rag",), ("hybrid",)}
+    assert valid_checkpoint(path, "case-1", "7-4", 3, expected_cells=expected)
+
+    rows[0]["rubric_details"] = {}
+    write_checkpoint(path, "7-4", rows)
+    assert not valid_checkpoint(path, "case-1", "7-4", 3, expected_cells=expected)
+
+    rows[0] = record(status="error")
+    write_checkpoint(path, "7-4", rows)
+    assert not valid_checkpoint(path, "case-1", "7-4", 3, expected_cells=expected)
+
+    rows = [record(system="rag") for _ in range(3)]
+    write_checkpoint(path, "7-4", rows)
+    assert not valid_checkpoint(path, "case-1", "7-4", 3, expected_cells=expected)
+
+
+def test_611_checkpoint_preserves_explicit_provider_errors(tmp_path):
+    path = tmp_path / "case.json"
+    rows = [record("7-11"), record("7-11", status="error")]
+    rows[1]["embedding"] = "blocked"
+    write_checkpoint(path, "7-11", rows)
+    assert valid_checkpoint(path, "case-1", "7-11", 2, {("e", "none", "m")})
+
+    rows[1]["error"] = None
+    write_checkpoint(path, "7-11", rows)
+    assert not valid_checkpoint(path, "case-1", "7-11", 2, {("e", "none", "m")})
+
+    rows[1]["error"] = "transient"
+    rows[1]["embedding"] = "e"
+    write_checkpoint(path, "7-11", rows)
+    assert not valid_checkpoint(path, "case-1", "7-11", 2, {("e", "none", "m")})
+
+
+def test_required_611_cells_excludes_only_preflight_failures():
+    config = {
+        "experiment_7_11": {
+            "embeddings": ["e-good", "e-bad"],
+            "rerankers": ["none", "r-bad"],
+            "main_models": ["m-good", "m-bad"],
+        }
+    }
+    readiness = {
+        "probes": [
+            {"component": "embedding", "name": "e-bad", "status": "error"},
+            {"component": "reranker", "name": "r-bad", "status": "error"},
+            {"component": "chat", "name": "m-bad", "status": "error"},
+        ]
+    }
+    assert required_611_cells(config, readiness) == {("e-good", "none", "m-good")}

chapter7/user-memory-system-evaluation/validation/verify_full_matrix_20260731.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+"""Honest verification of the merged Experiment 7-11 (README row 7-11) full matrix.
+
+Checks, without trusting the runner's own summary:
+  1. All 60 cases present, each with exactly 24 cells (4 embeddings x 3 rerankers x 2 main models).
+  2. 1,440 total records; zero error trajectories; zero unpriced requests/tokens.
+  3. Retrieval metrics (hit@5, recall@5, MRR) and task metrics (reward, success) are
+     populated and finite for every record.
+  4. Embedding-index cost accounting present for every record.
+  5. Interaction analysis: mean reward grouped by (embedding, reranker, main_model).
+
+Usage: python3 validation/verify_full_matrix_20260731.py [path-to-matrix.json]
+Exit 0 only if every hard check passes.
+"""
+
+import json
+import math
+import sys
+from collections import defaultdict
+
+PATH = sys.argv[1] if len(sys.argv) > 1 else "results/full_7_11_60_case_matrix.json"
+
+failures = []
+
+
+def check(cond, msg):
+    if not cond:
+        failures.append(msg)
+
+
+def main():
+    with open(PATH) as f:
+        data = json.load(f)
+
+    records = data.get("records", [])
+    by_case = defaultdict(list)
+    for r in records:
+        by_case[r["test_id"]].append(r)
+
+    # 1. coverage
+    check(len(by_case) == 60, f"expected 60 cases, got {len(by_case)}")
+    for tid, recs in sorted(by_case.items()):
+        check(len(recs) == 24, f"{tid}: expected 24 cells, got {len(recs)}")
+        combos = {(r["embedding"], r["reranker"], r["main_model"]) for r in recs}
+        check(len(combos) == 24, f"{tid}: duplicate/missing cell combos ({len(combos)} unique)")
+
+    # 2. totals and cleanliness
+    check(len(records) == 1440, f"expected 1440 records, got {len(records)}")
+    errors = [r for r in records if r.get("status") == "error" or r.get("error")]
+    check(not errors, f"{len(errors)} error trajectories")
+    unpriced_req = sum(r.get("unpriced_requests", 0) for r in records)
+    unpriced_tok = sum(r.get("unpriced_tokens", 0) for r in records)
+    check(unpriced_req == 0 and unpriced_tok == 0,
+          f"unpriced usage: {unpriced_req} requests, {unpriced_tok} tokens")
+
+    # 3. metrics populated
+    metric_fields = ["retrieval_hit_at_5", "retrieval_recall_at_5", "retrieval_mrr", "reward"]
+    for field in metric_fields:
+        bad = [r["test_id"] for r in records
+               if not isinstance(r.get(field), (int, float)) or not math.isfinite(r[field])]
+        check(not bad, f"metric {field} missing/non-finite in {len(bad)} records (e.g. {bad[:3]})")
+
+    # 4. embedding index cost accounting
+    no_idx = [r["test_id"] for r in records if r.get("embedding_index_latency_ms") is None]
+    check(not no_idx, f"embedding index accounting missing in {len(no_idx)} records")
+
+    # 5. interaction analysis (informational, always printed)
+    groups = defaultdict(list)
+    for r in records:
+        groups[(r["embedding"], r["reranker"], r["main_model"])].append(r["reward"])
+    print("mean reward by (embedding, reranker, main_model):")
+    for combo in sorted(groups):
+        vals = groups[combo]
+        print(f"  {combo}: {sum(vals)/len(vals):.4f} (n={len(vals)})")
+
+    total_cost = sum(r.get("cost_usd") or 0 for r in records)
+    print(f"total main-model cost: ${total_cost:.2f}")
+    print(f"records: {len(records)}, cases: {len(by_case)}")
+
+    if failures:
+        print("\nFAILURES:")
+        for msg in failures:
+            print(f"  - {msg}")
+        sys.exit(1)
+    print("\nALL CHECKS PASSED")
+
+
+if __name__ == "__main__":
+    main()