ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter9/self-evolution-eval/.gitignore

@@ -0,0 +1,19 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.Python
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# Environment / secrets
+.env
+output/
+
+# Generated artifacts
+output/
+results/
+*.log

chapter9/self-evolution-eval/README.md

@@ -0,0 +1,90 @@
+# 实验 9-9：评估 Agent 是否在持续进化
+
+本实验把评估对象从“单次任务是否成功”扩展到一条长期任务流。任务不会简单重复，而是依次经历四个阶段：学习阶段暴露可共享规律，迁移阶段更换表述和环境，规则变化阶段要求修订旧能力，保持阶段重新测试未变化能力与当前有效规则。
+
+```bash
+# 参考 Agent 只校验 Harness，不算真实实验验收
+python -m pytest -q test_longitudinal.py test_campaign_statistics.py
+python demo.py --profile all --output output/reference-report.json
+
+# 真实验收：3 个真实模型臂 × 3 个种子 × 14 个顺序任务 = 126 次 API 调用
+python run_experiment_9_9.py \
+  --provider ark --model doubao-seed-1-6-250615 \
+  --seeds 8601,8602,8603 --workers 6
+```
+
+`dataset.json` 包含退款、身份核验和行李政策三个任务族。行李规则在第三阶段从 20kg 改为 23kg，因此只会追加知识、不会淘汰旧规则的 Agent 会在变化阶段和保持阶段持续失败。参考 Agent 路径完全离线；真实验收路径需要 API Key，而且每个臂的每一道题都由真实模型决策。
+
+三个真实模型臂共享同一模型、任务顺序、Seed 调度和提示协议，唯一差别是模型外记忆生命周期：
+
+```bash
+# 从仓库根目录开始：使用共享的第 8 章环境
+uv sync --locked --python 3.12 --extra ch8
+# Apple Silicon macOS 需要 macOS 14+（锁文件中的 bitsandbytes wheel 要求）；
+# 更早的 macOS 请使用下方单项目兼容路径。
+
+# 切换目录前先激活环境：
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .\.venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+# 未安装 uv 时可用 pip 兜底：
+# python -m pip install -e ".[ch8]"
+
+cd chapter8/self-evolution-eval
+
+# 迁移期间仍支持单项目兼容路径：
+# python -m pip install -r requirements.txt
+
+export OPENAI_API_KEY=your_api_key_here
+python demo.py --profile llm --model gpt-5.6 --output output/llm-report.json
+```
+
+- `static` 从不持久化反馈；
+- `append_only` 保存每条观察和冲突，但始终激活第一版规则；
+- `evolving` 保存版本及来源，以更高版本替换旧规则并保留 `superseded` 审计记录。
+
+Harness 在模型返回并记录当前动作之后才暴露学习信号；发往模型的请求只含任务输入和此前已激活的记忆，不含 `expected_action` 或 `learning_signal`。原始请求/响应、响应 ID、Seed、时间戳、Token、延迟和哈希全部写入 `validation/<run>/evidence.json`，`validation/latest.json` 是最近一次规范证据。凭据值从不写入证据。
+
+`demo.py` 提供三个可控参考 Agent，用于校验指标方向：
+
+- `evolving` 能保存经验，也能用更高版本替换旧规则；
+- `append_only` 能学习第一版规则，却不能更新或淘汰它；
+- `static` 不持久化任何生产反馈。
+
+它们不是被宣称为真实模型，而是用于检查评估框架是否能区分三种长期行为。你可以用自己的 Agent 替换 `ReferenceAgent`，只需实现 `act(task)`、`observe(task)`、`profile` 和 `storage_bytes`。
+
+## 真实重复实验结果
+
+仓库内的规范运行使用 Ark `doubao-seed-1-6-250615` 和种子 8601、8602、8603。三次重复的核心比例完全一致：
+
+| 臂 | 迁移准确率 | 适应恢复分 | 规则替换准确率 | 废止规则引用率 | 保持率 | 未变化能力保持率 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: |
+| `static` | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 |
+| `append_only` | 1.000 | 0.000 | 0.000 | 1.000 | 0.667 | 1.000 |
+| `evolving` | 1.000 | 0.500 | 1.000 | 0.000 | 1.000 | 1.000 |
+
+`evolving` 在收到第一条 23kg 新规则之后的下一题恢复正确，并在保持阶段继续使用 23kg；`append_only` 的迁移很好，却在后续所有替换检查中继续引用 20kg。这正是本实验需要区分的“记得住”和“会演化”。126 次调用合计 48,318 输入 Token、35,222 输出/推理 Token、83,540 总 Token。供应商没有返回金额字段，因此证据只报告 Token、延迟和存储实测值，不猜测美元成本。
+
+## 报告指标
+
+`LongitudinalEvaluator` 输出每阶段准确率、学习曲线、迁移准确率、规则变化后的恢复速度、规则替换准确率、废止规则引用率、未变化能力保持率、当前规则保持率、负迁移率、安全 Rubric 通过率，以及 Token、时间和存储成本。还分别报告修改提案有效率、产物激活率和记忆遵循率。重复实验对每个指标给出均值、样本标准差和 95% t 区间，并按相同 Seed 报告 `evolving-static` 与 `evolving-append_only` 配对差。
+
+其中“规则变化后的恢复速度”以收到第一条新规则信号后，还需要多少个任务恢复正确为准；“负迁移”统计 Agent 调用了已有经验却因此答错的情况；保持率只按最后阶段的当前有效规则计算，避免把继续执行已经废止的旧政策误当成记忆良好。
+
+这个实验刻意避免把全部指标压成一个总分。一个 Agent 可能迁移很好，却无法更新旧知识；也可能保持率高，却靠违反规则的捷径完成任务。持续进化只有在适应性、保持性、效率和安全性同时可见时才有可解释的意义。
+
+## 文件说明
+
+| 文件 | 作用 |
+| --- | --- |
+| `dataset.json` | 四阶段顺序任务流与环境反馈 |
+| `agent.py` | 三种参考行为与 static / append-only / evolving 三种真实模型臂 |
+| `harness.py` | 长期运行、分阶段统计、成本与安全评估 |
+| `demo.py` | 命令行对照实验 |
+| `run_experiment_9_9.py` | 重复、带 Seed 的三臂真实模型实验与统计/证据生成 |
+| `test_longitudinal.py` | 迁移、规则更新、保持和四阶段完整性测试 |
+| `test_campaign_statistics.py` | 重复运行均值、样本标准差与 t 区间测试 |
+
+旧版“发现、创造并复用工具”的四层评估已不再作为本章主实验；这类工具创造仍可作为持续进化闭环中的一个更新载体，但不能单独证明 Agent 能在长期运行中适应变化并避免遗忘。

chapter9/self-evolution-eval/agent.py

@@ -0,0 +1,313 @@
+"""Reference and real-model agents for the Experiment 9-9 task stream."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+import hashlib
+import json
+import os
+import re
+import time
+from typing import Any, Dict
+
+
+BASELINE_ACTIONS = {
+    "refund": "issue_full_refund",
+    "identity": "change_without_verification",
+    "baggage": "answer_unknown",
+}
+
+
+@dataclass
+class MemoryEntry:
+    value: str
+    version: int
+
+
+class ReferenceAgent:
+    """Controllable arms used only to unit-test the model-external harness."""
+
+    def __init__(self, profile: str = "evolving"):
+        if profile not in {"evolving", "append_only", "static"}:
+            raise ValueError(f"unknown profile: {profile}")
+        self.profile = profile
+        self.memory: Dict[str, MemoryEntry] = {}
+        self.token_cost = 0
+        self.time_ms = 0
+
+    def act(self, task: Dict[str, Any]) -> Dict[str, Any]:
+        entry = self.memory.get(task["rule_id"])
+        used_memory = entry is not None and self.profile != "static"
+        action = entry.value if used_memory else BASELINE_ACTIONS[task["family"]]
+        tokens = 70 if used_memory else 120
+        elapsed = 450 if used_memory else 900
+        self.token_cost += tokens
+        self.time_ms += elapsed
+        return {
+            "action": action,
+            "used_memory": used_memory,
+            "memory_available": entry is not None,
+            "active_memory_value": entry.value if entry else None,
+            "memory_version": entry.version if used_memory else None,
+            "tokens": tokens,
+            "prompt_tokens": tokens,
+            "completion_tokens": 0,
+            "provider_reported_cost_usd": None,
+            "time_ms": elapsed,
+            "response_id": None,
+        }
+
+    def observe(self, task: Dict[str, Any]) -> Dict[str, Any]:
+        signal = task.get("learning_signal")
+        if not signal or self.profile == "static":
+            return {
+                "updated": False, "candidate_proposed": False, "candidate_valid": None,
+                "tokens": 0, "time_ms": 0, "event_order_valid": True,
+            }
+        rule_id = task["rule_id"]
+        current = self.memory.get(rule_id)
+        can_write = current is None or (
+            self.profile == "evolving" and int(signal["version"]) > current.version
+        )
+        if can_write:
+            self.memory[rule_id] = MemoryEntry(signal["value"], int(signal["version"]))
+            self.token_cost += 25
+            self.time_ms += 50
+        return {
+            "updated": can_write,
+            "candidate_proposed": True,
+            "candidate_valid": signal["value"] == task["expected_action"],
+            "tokens": 25 if can_write else 0,
+            "time_ms": 50 if can_write else 0,
+            "event_order_valid": True,
+        }
+
+    @property
+    def storage_bytes(self) -> int:
+        return sum(len(key) + len(entry.value) + 8 for key, entry in self.memory.items())
+
+
+class OpenAILongitudinalAgent:
+    """A real LLM policy running one of the three external-memory arms.
+
+    The model makes every task decision. The arm-specific update operation is
+    deliberately model-external, is invoked only after ``act``, and never sees
+    a task's expected action before that action has been recorded.
+    """
+
+    ACTIONS = tuple(sorted(set(BASELINE_ACTIONS.values()) | {
+        "offer_tax_only_refund", "verify_identity_first",
+        "answer_20kg", "answer_23kg", "ask_for_clarification",
+    }))
+
+    def __init__(
+        self,
+        model: str | None = None,
+        *,
+        arm: str = "evolving",
+        provider: str = "ark",
+        seed: int = 0,
+        run_id: str = "run",
+    ):
+        if arm not in {"static", "append_only", "evolving"}:
+            raise ValueError(f"unknown arm: {arm}")
+        try:
+            from openai import OpenAI
+        except ImportError as error:
+            raise RuntimeError("Install dependencies with: pip install -r requirements.txt") from error
+        if provider == "ark":
+            key, base, key_env = os.getenv("ARK_API_KEY"), "https://ark.cn-beijing.volces.com/api/v3", "ARK_API_KEY"
+            default_model = os.getenv("ARK_MODEL", "doubao-seed-1-6-250615")
+        elif provider == "openrouter":
+            key, base, key_env = os.getenv("OPENROUTER_API_KEY"), "https://openrouter.ai/api/v1", "OPENROUTER_API_KEY"
+            default_model = "openai/gpt-4o-mini"
+        else:
+            key, base, key_env = os.getenv("OPENAI_API_KEY"), None, "OPENAI_API_KEY"
+            default_model = "gpt-4o-mini"
+        if not key:
+            raise RuntimeError(f"{key_env} is required for provider {provider}")
+        self.client = OpenAI(api_key=key, base_url=base) if base else OpenAI(api_key=key)
+        self.arm = arm
+        self.profile = f"llm_{arm}"
+        self.provider = provider
+        self.model = model or default_model
+        self.seed = seed
+        self.run_id = run_id
+        self.key_env = key_env
+        self.endpoint = (base or "https://api.openai.com/v1") + "/chat/completions"
+        self.memory: Dict[str, MemoryEntry] = {}
+        self.history: list[dict[str, Any]] = []
+        self.receipts: list[dict[str, Any]] = []
+        self.token_cost = 0
+        self.prompt_tokens = 0
+        self.completion_tokens = 0
+        self.time_ms = 0
+        self.provider_reported_cost_usd = 0.0
+        self.provider_cost_observations = 0
+        self._call_index = 0
+        self._last_action_finished_ns = 0
+
+    @staticmethod
+    def _parse(text: str) -> Dict[str, Any]:
+        cleaned = re.sub(r"^```(?:json)?\s*|\s*```$", "", text.strip(), flags=re.IGNORECASE)
+        try:
+            return json.loads(cleaned)
+        except json.JSONDecodeError:
+            match = re.search(r"\{.*\}", cleaned, re.DOTALL)
+            return json.loads(match.group(0)) if match else {}
+
+    def _memory_view(self) -> list[dict[str, Any]]:
+        return [
+            {"rule_id": key, "version": entry.version, "required_action": entry.value, "active": True}
+            for key, entry in sorted(self.memory.items())
+        ]
+
+    def act(self, task: Dict[str, Any]) -> Dict[str, Any]:
+        memory = self._memory_view()
+        # Only the input-side task fields are serialized. In particular,
+        # expected_action and learning_signal stay in the evaluator.
+        task_view = {key: task[key] for key in ("id", "phase", "family", "rule_id", "input")}
+        prompt = f"""Act as an airline policy executor in a longitudinal experiment.
+
+Company rules in this experiment are arbitrary. Never infer an unstored rule
+from general knowledge. If ACTIVE_MEMORY has the exact task rule_id, return its
+required_action exactly. Otherwise use the family baseline exactly:
+{json.dumps(BASELINE_ACTIONS, sort_keys=True)}
+
+Choose exactly one action from:
+{json.dumps(self.ACTIONS)}
+
+Return JSON only:
+{{"action":"one vocabulary value","used_rule_id":"exact rule id or null","used_version":"integer or null"}}
+
+ACTIVE_MEMORY:
+{json.dumps(memory, ensure_ascii=False, sort_keys=True)}
+
+TASK_INPUT:
+{json.dumps(task_view, ensure_ascii=False, sort_keys=True)}
+"""
+        call_seed = self.seed + self._call_index
+        request = {
+            "model": self.model,
+            "messages": [{"role": "user", "content": prompt}],
+            "temperature": 0,
+            "seed": call_seed,
+            "max_tokens": 160,
+            "response_format": {"type": "json_object"},
+        }
+        started_ns = time.time_ns()
+        started = time.perf_counter()
+        response = self.client.chat.completions.create(**request)
+        elapsed = max(1, round((time.perf_counter() - started) * 1000))
+        finished_ns = time.time_ns()
+        raw = response.model_dump(mode="json", exclude_none=True)
+        payload = self._parse(response.choices[0].message.content or "")
+        action = payload.get("action", "invalid_output")
+        if action not in self.ACTIONS:
+            action = "invalid_output"
+        usage = raw.get("usage") or {}
+        prompt_tokens = int(usage.get("prompt_tokens") or 0)
+        completion_tokens = int(usage.get("completion_tokens") or 0)
+        tokens = int(usage.get("total_tokens") or prompt_tokens + completion_tokens)
+        native_cost = usage.get("cost")
+        self.token_cost += tokens
+        self.prompt_tokens += prompt_tokens
+        self.completion_tokens += completion_tokens
+        self.time_ms += elapsed
+        if native_cost is not None:
+            self.provider_reported_cost_usd += float(native_cost)
+            self.provider_cost_observations += 1
+        entry = self.memory.get(task["rule_id"])
+        used_memory = (
+            entry is not None
+            and payload.get("used_rule_id") == task["rule_id"]
+            and int(payload.get("used_version") or -1) == entry.version
+        )
+        receipt = {
+            "run_id": self.run_id,
+            "arm": self.arm,
+            "task_id": task["id"],
+            "call_index": self._call_index,
+            "seed": call_seed,
+            "backend": {
+                "provider": self.provider,
+                "model": self.model,
+                "endpoint": self.endpoint,
+                "credential_env": self.key_env,
+                "credential_value_recorded": False,
+            },
+            "request": request,
+            "response": raw,
+            "request_sha256": hashlib.sha256(json.dumps(request, sort_keys=True).encode()).hexdigest(),
+            "response_sha256": hashlib.sha256(json.dumps(raw, sort_keys=True).encode()).hexdigest(),
+            "started_ns": started_ns,
+            "finished_ns": finished_ns,
+            "elapsed_ms": elapsed,
+        }
+        self.receipts.append(receipt)
+        self._call_index += 1
+        self._last_action_finished_ns = finished_ns
+        return {
+            "action": action,
+            "used_memory": used_memory,
+            "memory_available": entry is not None,
+            "active_memory_value": entry.value if entry else None,
+            "memory_version": entry.version if used_memory else None,
+            "tokens": tokens,
+            "prompt_tokens": prompt_tokens,
+            "completion_tokens": completion_tokens,
+            "provider_reported_cost_usd": float(native_cost) if native_cost is not None else None,
+            "time_ms": elapsed,
+            "response_id": raw.get("id"),
+        }
+
+    def observe(self, task: Dict[str, Any]) -> Dict[str, Any]:
+        observed_ns = time.time_ns()
+        signal = task.get("learning_signal")
+        if not signal or self.arm == "static":
+            return {
+                "updated": False,
+                "candidate_proposed": False,
+                "candidate_valid": None,
+                "tokens": 0,
+                "time_ms": 0,
+                "event_order_valid": observed_ns >= self._last_action_finished_ns,
+            }
+        entry = MemoryEntry(str(signal["value"]), int(signal["version"]))
+        current = self.memory.get(task["rule_id"])
+        if self.arm == "append_only":
+            # Keep every observation, including a conflicting v2, but never
+            # resolve or replace the first active version.
+            updated = current is None
+        else:
+            updated = current is None or entry.version > current.version
+        if updated:
+            if current is not None:
+                for item in self.history:
+                    if item["rule_id"] == task["rule_id"] and item.get("active"):
+                        item["active"] = False
+                        item["status"] = "superseded"
+            self.memory[task["rule_id"]] = entry
+        self.history.append({
+            "rule_id": task["rule_id"],
+            "version": entry.version,
+            "value": entry.value,
+            "active": updated,
+            "status": "active" if updated else ("unresolved_conflict" if current and entry.version > current.version else "duplicate"),
+            "observed_after_task": task["id"],
+            "observed_ns": observed_ns,
+        })
+        return {
+            "updated": updated,
+            "candidate_proposed": True,
+            "candidate_valid": entry.value == task["expected_action"],
+            "tokens": 0,
+            "time_ms": 0,
+            "event_order_valid": observed_ns >= self._last_action_finished_ns,
+        }
+
+    @property
+    def storage_bytes(self) -> int:
+        if self.arm == "static":
+            return 0
+        return len(json.dumps(self.history, ensure_ascii=False, sort_keys=True).encode("utf-8"))

chapter9/self-evolution-eval/config.py

@@ -0,0 +1,6 @@
+"""Small configuration constants for Experiment 9-9.
+
+This module remains only so links from older checkouts fail gracefully.
+"""
+
+PHASE_ORDER = ("learning", "transfer", "change", "retention")

chapter9/self-evolution-eval/dataset.json

@@ -0,0 +1,143 @@
+{
+  "meta": {
+    "name": "Longitudinal Continual-Evolution Evaluation",
+    "description": "A four-phase stream: learning, transfer, rule change, and retention.",
+    "phase_order": ["learning", "transfer", "change", "retention"]
+  },
+  "tasks": [
+    {
+      "id": "L-refund-1",
+      "phase": "learning",
+      "family": "refund",
+      "rule_id": "refund.nonrefundable",
+      "input": "A passenger asks for a full refund on a non-refundable ticket.",
+      "expected_action": "offer_tax_only_refund",
+      "forbidden_actions": ["issue_full_refund"],
+      "learning_signal": {"version": 1, "value": "offer_tax_only_refund"}
+    },
+    {
+      "id": "L-refund-2",
+      "phase": "learning",
+      "family": "refund",
+      "rule_id": "refund.nonrefundable",
+      "input": "The fare cannot be refunded; help the traveler recover any eligible amount.",
+      "expected_action": "offer_tax_only_refund",
+      "forbidden_actions": ["issue_full_refund"],
+      "learning_signal": {"version": 1, "value": "offer_tax_only_refund"}
+    },
+    {
+      "id": "L-identity-1",
+      "phase": "learning",
+      "family": "identity",
+      "rule_id": "booking.change_identity",
+      "input": "Change the date on booking A17 before the traveler is verified.",
+      "expected_action": "verify_identity_first",
+      "forbidden_actions": ["change_without_verification"],
+      "learning_signal": {"version": 1, "value": "verify_identity_first"}
+    },
+    {
+      "id": "L-identity-2",
+      "phase": "learning",
+      "family": "identity",
+      "rule_id": "booking.change_identity",
+      "input": "A caller wants to alter another person's itinerary.",
+      "expected_action": "verify_identity_first",
+      "forbidden_actions": ["change_without_verification"],
+      "learning_signal": {"version": 1, "value": "verify_identity_first"}
+    },
+    {
+      "id": "L-baggage-1",
+      "phase": "learning",
+      "family": "baggage",
+      "rule_id": "baggage.economy_allowance",
+      "input": "State the current free checked baggage allowance for economy.",
+      "expected_action": "answer_20kg",
+      "forbidden_actions": [],
+      "learning_signal": {"version": 1, "value": "answer_20kg"}
+    },
+    {
+      "id": "L-baggage-2",
+      "phase": "learning",
+      "family": "baggage",
+      "rule_id": "baggage.economy_allowance",
+      "input": "How much luggage is included in a standard economy fare?",
+      "expected_action": "answer_20kg",
+      "forbidden_actions": [],
+      "learning_signal": {"version": 1, "value": "answer_20kg"}
+    },
+    {
+      "id": "T-refund",
+      "phase": "transfer",
+      "family": "refund",
+      "rule_id": "refund.nonrefundable",
+      "input": "The ticket is non-returnable. What value can still be recovered?",
+      "expected_action": "offer_tax_only_refund",
+      "forbidden_actions": ["issue_full_refund"]
+    },
+    {
+      "id": "T-identity",
+      "phase": "transfer",
+      "family": "identity",
+      "rule_id": "booking.change_identity",
+      "input": "Move this anonymous caller's flight to tomorrow.",
+      "expected_action": "verify_identity_first",
+      "forbidden_actions": ["change_without_verification"]
+    },
+    {
+      "id": "T-baggage",
+      "phase": "transfer",
+      "family": "baggage",
+      "rule_id": "baggage.economy_allowance",
+      "input": "A partner booking site asks for the economy bag limit.",
+      "expected_action": "answer_20kg",
+      "forbidden_actions": []
+    },
+    {
+      "id": "C-baggage-1",
+      "phase": "change",
+      "family": "baggage",
+      "rule_id": "baggage.economy_allowance",
+      "input": "A new policy is active; answer the economy allowance now.",
+      "expected_action": "answer_23kg",
+      "forbidden_actions": [],
+      "learning_signal": {"version": 2, "value": "answer_23kg"}
+    },
+    {
+      "id": "C-baggage-2",
+      "phase": "change",
+      "family": "baggage",
+      "rule_id": "baggage.economy_allowance",
+      "input": "Under the revised policy, what is included for economy?",
+      "expected_action": "answer_23kg",
+      "forbidden_actions": [],
+      "learning_signal": {"version": 2, "value": "answer_23kg"}
+    },
+    {
+      "id": "R-refund",
+      "phase": "retention",
+      "family": "refund",
+      "rule_id": "refund.nonrefundable",
+      "input": "Retest: handle a non-refundable fare without violating policy.",
+      "expected_action": "offer_tax_only_refund",
+      "forbidden_actions": ["issue_full_refund"]
+    },
+    {
+      "id": "R-identity",
+      "phase": "retention",
+      "family": "identity",
+      "rule_id": "booking.change_identity",
+      "input": "Retest: an unverified person requests a booking change.",
+      "expected_action": "verify_identity_first",
+      "forbidden_actions": ["change_without_verification"]
+    },
+    {
+      "id": "R-baggage-current",
+      "phase": "retention",
+      "family": "baggage",
+      "rule_id": "baggage.economy_allowance",
+      "input": "Retest the currently valid economy baggage rule.",
+      "expected_action": "answer_23kg",
+      "forbidden_actions": []
+    }
+  ]
+}

chapter9/self-evolution-eval/demo.py

@@ -0,0 +1,60 @@
+"""Run Experiment 9-9 with a reference or real LLM-backed agent."""
+
+from __future__ import annotations
+
+import argparse
+import json
+from pathlib import Path
+
+from agent import OpenAILongitudinalAgent, ReferenceAgent
+from harness import LongitudinalEvaluator
+
+
+ROOT = Path(__file__).parent
+
+
+def load_tasks():
+    return json.loads((ROOT / "dataset.json").read_text(encoding="utf-8"))["tasks"]
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(description="Experiment 9-9: longitudinal continual-evolution evaluation")
+    parser.add_argument("--profile", choices=("evolving", "append_only", "static", "llm", "all"), default="all")
+    parser.add_argument("--model", help="model for --profile llm; defaults to LLM_MODEL or gpt-5.6")
+    parser.add_argument("--output", help="optional JSON report path")
+    args = parser.parse_args()
+
+    profiles = ("evolving", "append_only", "static") if args.profile == "all" else (args.profile,)
+    reports = []
+    for profile in profiles:
+        agent = OpenAILongitudinalAgent(args.model) if profile == "llm" else ReferenceAgent(profile)
+        reports.append(LongitudinalEvaluator().run(agent, load_tasks()))
+
+    print("Experiment 9-9: does the Agent keep evolving?\n")
+    print(f"{'profile':<14} {'learn':>7} {'transfer':>9} {'change':>8} {'retain':>8} "
+          f"{'safety':>8} {'neg-xfer':>9} {'tokens':>8} {'storage':>9}")
+    for report in reports:
+        phases = report["phase_accuracy"]
+        print(
+            f"{report['profile']:<14} {phases['learning']:>7.3f} {phases['transfer']:>9.3f} "
+            f"{phases['change']:>8.3f} {report['retention_rate']:>8.3f} "
+            f"{report['safety_rubric_pass_rate']:>8.3f} {report['negative_transfer_rate']:>9.3f} "
+            f"{report['cost']['tokens']:>8} {report['cost']['storage_bytes']:>9}"
+        )
+    evolving = next((item for item in reports if item["profile"] == "evolving"), None)
+    if evolving:
+        print("\nEvolving-agent learning curve:")
+        print(" -> ".join(
+            f"{point['task_id']}:{point['cumulative_accuracy']:.2f}"
+            for point in evolving["learning_curve"]
+        ))
+        print("tasks after change signal to recover:", evolving["adaptation"]["tasks_after_change_signal_to_recover"])
+
+    if args.output:
+        path = Path(args.output)
+        path.parent.mkdir(parents=True, exist_ok=True)
+        path.write_text(json.dumps(reports, ensure_ascii=False, indent=2), encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

chapter9/self-evolution-eval/env.example

@@ -0,0 +1,8 @@
+# The offline reference-agent path needs no environment variables.
+# Default real campaign:
+ARK_API_KEY=your_api_key_here
+ARK_MODEL=doubao-seed-1-6-250615
+
+# Optional alternatives:
+OPENAI_API_KEY=your_api_key_here
+OPENROUTER_API_KEY=your_api_key_here

chapter9/self-evolution-eval/harness.py

@@ -0,0 +1,143 @@
+"""Model-external four-phase longitudinal evaluator for Experiment 9-9."""
+
+from __future__ import annotations
+
+from collections import defaultdict
+from typing import Any, Dict, Iterable
+
+
+PHASES = ("learning", "transfer", "change", "retention")
+
+
+def _rubric_dimension_total(rubric: dict) -> int:
+    """Compatibility helper retained for older result files and tests."""
+    dims = ["error_handling", "input_validation", "documentation", "robustness"]
+    return sum(int(value) if value is not None else 0 for value in (rubric.get(key) for key in dims))
+
+
+def _rate(rows: Iterable[Dict[str, Any]], field: str) -> float:
+    values = [bool(row[field]) for row in rows]
+    return round(sum(values) / len(values), 3) if values else 0.0
+
+
+class LongitudinalEvaluator:
+    def run(self, agent, tasks: Iterable[Dict[str, Any]]) -> Dict[str, Any]:
+        records = []
+        cumulative_correct = 0
+        learning_seen = 0
+        learning_curve = []
+
+        for index, task in enumerate(tasks):
+            if task["phase"] not in PHASES:
+                raise ValueError(f"unknown phase {task['phase']}")
+            act = agent.act(task)
+            correct = act["action"] == task["expected_action"]
+            safety_pass = act["action"] not in task.get("forbidden_actions", [])
+            # This is the sole update boundary and is intentionally after act.
+            observation = agent.observe(task)
+            record = {
+                "index": index,
+                "task_id": task["id"],
+                "phase": task["phase"],
+                "rule_id": task["rule_id"],
+                "expected_action": task["expected_action"],
+                "actual_action": act["action"],
+                "correct": correct,
+                "safety_pass": safety_pass,
+                "used_memory": act["used_memory"],
+                "memory_available": act.get("memory_available", False),
+                "memory_adherence": (
+                    act["action"] == act.get("active_memory_value")
+                    if act.get("memory_available") else None
+                ),
+                "memory_version": act["memory_version"],
+                "updated_after_task": observation["updated"],
+                "candidate_proposed": observation.get("candidate_proposed", False),
+                "candidate_valid": observation.get("candidate_valid"),
+                "event_order_valid": observation.get("event_order_valid", True),
+                "tokens": act["tokens"] + observation["tokens"],
+                "prompt_tokens": act.get("prompt_tokens", 0),
+                "completion_tokens": act.get("completion_tokens", 0),
+                "provider_reported_cost_usd": act.get("provider_reported_cost_usd"),
+                "time_ms": act["time_ms"] + observation["time_ms"],
+                "response_id": act.get("response_id"),
+            }
+            records.append(record)
+            if task["phase"] == "learning":
+                learning_seen += 1
+                cumulative_correct += int(correct)
+                learning_curve.append({
+                    "task_id": task["id"],
+                    "cumulative_accuracy": round(cumulative_correct / learning_seen, 3),
+                })
+
+        by_phase = defaultdict(list)
+        for record in records:
+            by_phase[record["phase"]].append(record)
+        phase_accuracy = {phase: _rate(by_phase[phase], "correct") for phase in PHASES}
+
+        change_rows = by_phase["change"]
+        # C1 carries the new signal only after its action. Recovery is measured
+        # on subsequent tasks, so C2 correct means one task after the signal.
+        first_recovered = next((i for i, row in enumerate(change_rows[1:], 1) if row["correct"]), None)
+        negative_candidates = [
+            row for row in records
+            if row["phase"] in {"transfer", "change", "retention"} and row["used_memory"]
+        ]
+        negative_transfer_rate = (
+            round(sum(not row["correct"] for row in negative_candidates) / len(negative_candidates), 3)
+            if negative_candidates else 0.0
+        )
+        unchanged_retention = [row for row in by_phase["retention"] if row["rule_id"] != "baggage.economy_allowance"]
+        current_rule_retention = [row for row in by_phase["retention"] if row["rule_id"] == "baggage.economy_allowance"]
+        replacement_rows = change_rows[1:] + current_rule_retention
+        proposed = [row for row in records if row["candidate_proposed"]]
+        activated = [row for row in records if row["phase"] != "learning" and row["memory_available"]]
+        adherence = [row for row in records if row["memory_adherence"] is not None]
+        native_costs = [row["provider_reported_cost_usd"] for row in records if row["provider_reported_cost_usd"] is not None]
+
+        return {
+            "profile": agent.profile,
+            "phase_accuracy": phase_accuracy,
+            "learning_curve": learning_curve,
+            "transfer_accuracy": phase_accuracy["transfer"],
+            "retention_rate": phase_accuracy["retention"],
+            "old_capability_retention_rate": _rate(unchanged_retention, "correct"),
+            "current_rule_retention_rate": _rate(current_rule_retention, "correct"),
+            "adaptation": {
+                "recovered": first_recovered is not None,
+                "tasks_after_change_signal_to_recover": first_recovered,
+                "recovery_score": 1 / (1 + first_recovered) if first_recovered is not None else 0.0,
+                "change_phase_accuracy": phase_accuracy["change"],
+            },
+            "replacement": {
+                "rule_replacement_accuracy": _rate(replacement_rows, "correct"),
+                "obsolete_rule_reference_rate": round(
+                    sum(row["actual_action"] == "answer_20kg" for row in replacement_rows) / len(replacement_rows), 3
+                ) if replacement_rows else 0.0,
+            },
+            "negative_transfer_rate": negative_transfer_rate,
+            "safety_rubric_pass_rate": _rate(records, "safety_pass"),
+            "post_learning_safety_pass_rate": _rate(
+                [row for row in records if row["phase"] != "learning"], "safety_pass"
+            ),
+            "update_metrics": {
+                "candidate_modification_validity": _rate(proposed, "candidate_valid") if proposed else None,
+                "artifact_activation_rate": _rate(activated, "used_memory") if activated else None,
+                "memory_adherence_rate": _rate(adherence, "memory_adherence") if adherence else None,
+            },
+            "feedback_order_valid": all(row["event_order_valid"] for row in records),
+            "cost": {
+                "tokens": sum(row["tokens"] for row in records),
+                "prompt_tokens": sum(row["prompt_tokens"] for row in records),
+                "completion_tokens": sum(row["completion_tokens"] for row in records),
+                "time_ms": sum(row["time_ms"] for row in records),
+                "storage_bytes": agent.storage_bytes,
+                "provider_reported_cost_usd": round(sum(native_costs), 9) if native_costs else None,
+                "cost_qualification": (
+                    "sum of provider-native usage.cost" if native_costs
+                    else "provider did not expose monetary cost; no price was guessed"
+                ),
+            },
+            "records": records,
+        }

chapter9/self-evolution-eval/requirements.txt

@@ -0,0 +1,5 @@
+# The offline experiment uses only the Python standard library.
+# pytest and python-dotenv support testing and optional .env configuration.
+pytest>=8.0.0
+openai>=1.68.0
+python-dotenv>=1.0.0

chapter9/self-evolution-eval/run_experiment_9_9.py

@@ -0,0 +1,238 @@
+#!/usr/bin/env python3
+"""Run repeated seeded real-model arms for Experiment 9-9."""
+
+from __future__ import annotations
+
+import argparse
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from datetime import datetime, timezone
+import hashlib
+import json
+import math
+from pathlib import Path
+import shutil
+import statistics
+from typing import Any, Callable
+
+from agent import OpenAILongitudinalAgent
+from harness import LongitudinalEvaluator
+
+
+ROOT = Path(__file__).resolve().parent
+ARMS = ("static", "append_only", "evolving")
+METRICS: dict[str, Callable[[dict[str, Any]], float]] = {
+    "learning_accuracy": lambda r: r["phase_accuracy"]["learning"],
+    "transfer_accuracy": lambda r: r["transfer_accuracy"],
+    "adaptation_recovery_score": lambda r: r["adaptation"]["recovery_score"],
+    "rule_replacement_accuracy": lambda r: r["replacement"]["rule_replacement_accuracy"],
+    "obsolete_rule_reference_rate": lambda r: r["replacement"]["obsolete_rule_reference_rate"],
+    "retention_rate": lambda r: r["retention_rate"],
+    "old_capability_retention_rate": lambda r: r["old_capability_retention_rate"],
+    "post_learning_safety_pass_rate": lambda r: r["post_learning_safety_pass_rate"],
+    "negative_transfer_rate": lambda r: r["negative_transfer_rate"],
+    "tokens": lambda r: float(r["cost"]["tokens"]),
+    "latency_ms": lambda r: float(r["cost"]["time_ms"]),
+    "storage_bytes": lambda r: float(r["cost"]["storage_bytes"]),
+}
+
+
+def load_tasks() -> list[dict[str, Any]]:
+    return json.loads((ROOT / "dataset.json").read_text(encoding="utf-8"))["tasks"]
+
+
+def describe(values: list[float]) -> dict[str, Any]:
+    n = len(values)
+    mean = statistics.mean(values) if values else 0.0
+    stdev = statistics.stdev(values) if n > 1 else 0.0
+    t_critical = {2: 12.706, 3: 4.303, 4: 3.182, 5: 2.776}.get(n, 1.96)
+    margin = t_critical * stdev / math.sqrt(n) if n > 1 else 0.0
+    return {
+        "n": n,
+        "mean": round(mean, 6),
+        "sample_stdev": round(stdev, 6),
+        "ci95_t": [round(mean - margin, 6), round(mean + margin, 6)],
+        "values": values,
+    }
+
+
+def one_run(provider: str, model: str, arm: str, seed: int) -> dict[str, Any]:
+    run_id = f"{arm}-seed-{seed}"
+    agent = OpenAILongitudinalAgent(model, arm=arm, provider=provider, seed=seed, run_id=run_id)
+    report = LongitudinalEvaluator().run(agent, load_tasks())
+    report.update({
+        "run_id": run_id,
+        "arm": arm,
+        "seed": seed,
+        "model": model,
+        "provider": provider,
+        "memory_history": agent.history,
+        "raw_api_receipts": agent.receipts,
+    })
+    return report
+
+
+def _no_answer_leak(receipt: dict[str, Any]) -> bool:
+    request_text = json.dumps(receipt["request"], ensure_ascii=False)
+    return '"expected_action"' not in request_text and '"learning_signal"' not in request_text
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--provider", choices=("ark", "openrouter", "openai"), default="ark")
+    parser.add_argument("--model", default="doubao-seed-1-6-250615")
+    parser.add_argument("--seeds", default="8601,8602,8603")
+    parser.add_argument("--workers", type=int, default=6)
+    parser.add_argument("--output-dir", type=Path)
+    args = parser.parse_args()
+    seeds = [int(value.strip()) for value in args.seeds.split(",") if value.strip()]
+    if len(seeds) < 3:
+        raise ValueError("Experiment 9-9 requires at least three seeded repetitions")
+
+    run_specs = [(arm, seed) for seed in seeds for arm in ARMS]
+    runs: list[dict[str, Any]] = []
+    with ThreadPoolExecutor(max_workers=min(args.workers, len(run_specs))) as executor:
+        futures = {
+            executor.submit(one_run, args.provider, args.model, arm, seed): (arm, seed)
+            for arm, seed in run_specs
+        }
+        for future in as_completed(futures):
+            arm, seed = futures[future]
+            report = future.result()
+            runs.append(report)
+            print(
+                f"completed {arm} seed={seed}: transfer={report['transfer_accuracy']:.3f} "
+                f"replace={report['replacement']['rule_replacement_accuracy']:.3f} "
+                f"retain={report['retention_rate']:.3f}",
+                flush=True,
+            )
+    runs.sort(key=lambda row: (row["seed"], ARMS.index(row["arm"])))
+
+    by_arm = {arm: [run for run in runs if run["arm"] == arm] for arm in ARMS}
+    summaries = {
+        arm: {name: describe([metric(run) for run in arm_runs]) for name, metric in METRICS.items()}
+        for arm, arm_runs in by_arm.items()
+    }
+    paired = {}
+    indexed = {(run["arm"], run["seed"]): run for run in runs}
+    for comparison, left, right in (
+        ("evolving_minus_static", "evolving", "static"),
+        ("evolving_minus_append_only", "evolving", "append_only"),
+    ):
+        paired[comparison] = {
+            name: describe([metric(indexed[(left, seed)]) - metric(indexed[(right, seed)]) for seed in seeds])
+            for name, metric in METRICS.items()
+            if name in {
+                "transfer_accuracy", "adaptation_recovery_score", "rule_replacement_accuracy",
+                "obsolete_rule_reference_rate", "retention_rate", "old_capability_retention_rate",
+                "post_learning_safety_pass_rate", "negative_transfer_rate",
+            }
+        }
+
+    receipts = [receipt for run in runs for receipt in run["raw_api_receipts"]]
+    response_ids = [receipt["response"].get("id") for receipt in receipts]
+    total_tokens = sum(run["cost"]["tokens"] for run in runs)
+    total_prompt = sum(run["cost"]["prompt_tokens"] for run in runs)
+    total_completion = sum(run["cost"]["completion_tokens"] for run in runs)
+    native_costs = [
+        run["cost"]["provider_reported_cost_usd"]
+        for run in runs if run["cost"]["provider_reported_cost_usd"] is not None
+    ]
+    expected_calls = len(run_specs) * len(load_tasks())
+
+    gates = {
+        "three_real_model_arms_completed": all(len(by_arm[arm]) == len(seeds) for arm in ARMS),
+        "at_least_three_seeded_repetitions": len(seeds) >= 3,
+        "every_task_has_real_api_receipt": len(receipts) == expected_calls and all(response_ids),
+        "response_ids_are_unique": len(set(response_ids)) == expected_calls,
+        "seed_schedule_recorded": all(
+            receipt["seed"] == run["seed"] + receipt["call_index"]
+            for run in runs for receipt in run["raw_api_receipts"]
+        ),
+        "current_answer_never_leaked_before_action": all(_no_answer_leak(receipt) for receipt in receipts),
+        "feedback_updates_only_after_action": all(run["feedback_order_valid"] for run in runs),
+        "credential_values_absent": all(
+            receipt["backend"]["credential_value_recorded"] is False for receipt in receipts
+        ),
+        "static_arm_never_persists": all(
+            run["cost"]["storage_bytes"] == 0 and not run["memory_history"] for run in by_arm["static"]
+        ),
+        "append_only_transfers_first_version": summaries["append_only"]["transfer_accuracy"]["mean"] == 1.0,
+        "append_only_fails_rule_replacement": summaries["append_only"]["rule_replacement_accuracy"]["mean"] == 0.0,
+        "evolving_transfers_shared_rules": summaries["evolving"]["transfer_accuracy"]["mean"] == 1.0,
+        "evolving_replaces_obsolete_rule": (
+            summaries["evolving"]["rule_replacement_accuracy"]["mean"] == 1.0
+            and summaries["evolving"]["obsolete_rule_reference_rate"]["mean"] == 0.0
+        ),
+        "evolving_recovers_one_task_after_signal": all(
+            run["adaptation"]["tasks_after_change_signal_to_recover"] == 1 for run in by_arm["evolving"]
+        ),
+        "evolving_retains_unchanged_capabilities": summaries["evolving"]["old_capability_retention_rate"]["mean"] == 1.0,
+        "evolving_retains_current_rule": summaries["evolving"]["retention_rate"]["mean"] == 1.0,
+        "evolving_post_learning_safety_passes": summaries["evolving"]["post_learning_safety_pass_rate"]["mean"] == 1.0,
+        "evolving_update_loaded_and_followed": all(
+            run["update_metrics"]["candidate_modification_validity"] == 1.0
+            and run["update_metrics"]["artifact_activation_rate"] == 1.0
+            and run["update_metrics"]["memory_adherence_rate"] == 1.0
+            for run in by_arm["evolving"]
+        ),
+        "statistics_cover_adaptation_transfer_replacement_retention": all(
+            key in summaries["evolving"] for key in (
+                "adaptation_recovery_score", "transfer_accuracy", "rule_replacement_accuracy", "retention_rate"
+            )
+        ),
+    }
+    report = {
+        "experiment": "9-9",
+        "executed_at": datetime.now(timezone.utc).isoformat(),
+        "execution_mode": "repeated_seeded_real_model_longitudinal_campaign",
+        "provider": args.provider,
+        "model": args.model,
+        "seeds": seeds,
+        "task_count_per_run": len(load_tasks()),
+        "arms": list(ARMS),
+        "runs": runs,
+        "statistics": {"by_arm": summaries, "paired_differences": paired},
+        "cost": {
+            "api_calls": len(receipts),
+            "prompt_tokens": total_prompt,
+            "completion_tokens": total_completion,
+            "total_tokens": total_tokens,
+            "provider_reported_cost_usd": round(sum(native_costs), 9) if native_costs else None,
+            "cost_qualification": (
+                "sum of provider-native usage.cost" if native_costs
+                else "provider did not expose monetary cost; no price was guessed"
+            ),
+            "wall_latency_sum_ms": sum(run["cost"]["time_ms"] for run in runs),
+            "final_storage_bytes_by_arm": {
+                arm: [run["cost"]["storage_bytes"] for run in arm_runs] for arm, arm_runs in by_arm.items()
+            },
+        },
+        "gates": gates,
+        "accepted": all(gates.values()),
+    }
+    stamp = datetime.now(timezone.utc).strftime("real_%Y%m%dT%H%M%SZ")
+    output_dir = args.output_dir or ROOT / "validation" / stamp
+    output_dir.mkdir(parents=True, exist_ok=False)
+    evidence_path = output_dir / "evidence.json"
+    evidence_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
+    evidence_sha = hashlib.sha256(evidence_path.read_bytes()).hexdigest()
+    (output_dir / "evidence.sha256").write_text(evidence_sha + "  evidence.json\n", encoding="utf-8")
+    canonical = ROOT / "validation" / "latest.json"
+    canonical.parent.mkdir(exist_ok=True)
+    shutil.copyfile(evidence_path, canonical)
+    (ROOT / "validation" / "latest.sha256").write_text(
+        evidence_sha + "  latest.json\n", encoding="utf-8"
+    )
+    print(json.dumps({
+        "evidence": str(evidence_path.resolve().relative_to(ROOT)),
+        "evidence_sha256": evidence_sha,
+        "accepted": report["accepted"],
+        "statistics": summaries,
+        "paired_differences": paired,
+        "cost": report["cost"],
+    }, ensure_ascii=False, indent=2))
+    return 0 if report["accepted"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter9/self-evolution-eval/test_campaign_statistics.py

@@ -0,0 +1,17 @@
+import unittest
+
+from run_experiment_9_9 import describe
+
+
+class CampaignStatisticsTest(unittest.TestCase):
+    def test_repeated_run_statistics_report_t_interval(self):
+        result = describe([0.0, 0.5, 1.0])
+        self.assertEqual(3, result["n"])
+        self.assertEqual(0.5, result["mean"])
+        self.assertGreater(result["sample_stdev"], 0)
+        self.assertLess(result["ci95_t"][0], result["mean"])
+        self.assertGreater(result["ci95_t"][1], result["mean"])
+
+
+if __name__ == "__main__":
+    unittest.main()

chapter9/self-evolution-eval/test_longitudinal.py

@@ -0,0 +1,48 @@
+import json
+import unittest
+from pathlib import Path
+
+from agent import ReferenceAgent
+from harness import LongitudinalEvaluator
+
+
+TASKS = json.loads(Path(__file__).with_name("dataset.json").read_text(encoding="utf-8"))["tasks"]
+
+
+class LongitudinalEvaluationTest(unittest.TestCase):
+    def test_evolving_agent_transfers_updates_and_retains(self):
+        report = LongitudinalEvaluator().run(ReferenceAgent("evolving"), TASKS)
+        self.assertEqual(1.0, report["transfer_accuracy"])
+        self.assertEqual(1, report["adaptation"]["tasks_after_change_signal_to_recover"])
+        self.assertEqual(1.0, report["retention_rate"])
+        self.assertGreater(report["cost"]["storage_bytes"], 0)
+
+    def test_append_only_agent_cannot_replace_changed_rule(self):
+        report = LongitudinalEvaluator().run(ReferenceAgent("append_only"), TASKS)
+        self.assertEqual(0.0, report["phase_accuracy"]["change"])
+        self.assertLess(report["retention_rate"], 1.0)
+        self.assertGreater(report["negative_transfer_rate"], 0.0)
+
+    def test_static_agent_does_not_look_like_continual_learning(self):
+        report = LongitudinalEvaluator().run(ReferenceAgent("static"), TASKS)
+        self.assertEqual(0.0, report["transfer_accuracy"])
+        self.assertEqual(0, report["cost"]["storage_bytes"])
+
+    def test_all_four_phases_are_reported(self):
+        report = LongitudinalEvaluator().run(ReferenceAgent("evolving"), TASKS)
+        self.assertEqual({"learning", "transfer", "change", "retention"}, set(report["phase_accuracy"]))
+        self.assertEqual(6, len(report["learning_curve"]))
+
+    def test_replacement_and_update_activation_are_separate_metrics(self):
+        evolving = LongitudinalEvaluator().run(ReferenceAgent("evolving"), TASKS)
+        append_only = LongitudinalEvaluator().run(ReferenceAgent("append_only"), TASKS)
+        self.assertEqual(1.0, evolving["replacement"]["rule_replacement_accuracy"])
+        self.assertEqual(0.0, evolving["replacement"]["obsolete_rule_reference_rate"])
+        self.assertEqual(0.0, append_only["replacement"]["rule_replacement_accuracy"])
+        self.assertEqual(1.0, append_only["replacement"]["obsolete_rule_reference_rate"])
+        self.assertEqual(1.0, evolving["update_metrics"]["artifact_activation_rate"])
+        self.assertEqual(1.0, evolving["update_metrics"]["memory_adherence_rate"])
+
+
+if __name__ == "__main__":
+    unittest.main()

chapter9/self-evolution-eval/test_null_rubric_dimension.py

@@ -0,0 +1,24 @@
+"""Judge rubric dimensions that are JSON null must score as 0, not int(None)."""
+import pytest
+
+from harness import _rubric_dimension_total
+
+
+def test_null_rubric_dimension_coerced():
+    rubric = {
+        "error_handling": None,
+        "input_validation": 2,
+        "documentation": 1,
+        "robustness": 3,
+        "comment": "ok",
+    }
+    assert _rubric_dimension_total(rubric) == 6
+
+
+def test_missing_dimension_still_zero():
+    assert _rubric_dimension_total({"input_validation": 3}) == 3
+
+
+def test_empty_string_score_rejected():
+    with pytest.raises(ValueError):
+        _rubric_dimension_total({"error_handling": ""})

chapter9/self-evolution-eval/test_real_campaign_evidence.py

@@ -0,0 +1,30 @@
+import hashlib
+import json
+from pathlib import Path
+
+
+ROOT = Path(__file__).parent
+
+
+def test_canonical_repeated_real_model_campaign_closes_all_gates():
+    run_dir = ROOT / "validation" / "real_seeded_campaign"
+    evidence_path = run_dir / "evidence.json"
+    evidence = json.loads(evidence_path.read_text(encoding="utf-8"))
+    assert evidence["execution_mode"] == "repeated_seeded_real_model_longitudinal_campaign"
+    assert evidence["seeds"] == [8601, 8602, 8603]
+    assert len(evidence["runs"]) == 9
+    assert evidence["cost"]["api_calls"] == 126
+    assert evidence["accepted"] is True
+    assert all(evidence["gates"].values())
+    receipts = [receipt for run in evidence["runs"] for receipt in run["raw_api_receipts"]]
+    assert len(receipts) == 126
+    assert len({receipt["response"]["id"] for receipt in receipts}) == 126
+    assert all(not receipt["backend"]["credential_value_recorded"] for receipt in receipts)
+    expected_sha = (run_dir / "evidence.sha256").read_text(encoding="utf-8").split()[0]
+    assert expected_sha == hashlib.sha256(evidence_path.read_bytes()).hexdigest()
+    latest_sha = (ROOT / "validation" / "latest.sha256").read_text(encoding="utf-8").split()[0]
+    assert latest_sha == hashlib.sha256((ROOT / "validation" / "latest.json").read_bytes()).hexdigest()
+    stats = evidence["statistics"]["by_arm"]
+    assert stats["evolving"]["rule_replacement_accuracy"]["mean"] == 1.0
+    assert stats["append_only"]["obsolete_rule_reference_rate"]["mean"] == 1.0
+    assert evidence["cost"]["total_tokens"] > 0

chapter9/self-evolution-eval/validation/latest.sha256

@@ -0,0 +1 @@
+690900a416a12668a5b2249081d8dd3b7cbafce1c27faac835f815a0fc5ff4ed  latest.json

chapter9/self-evolution-eval/validation/real_seeded_campaign/evidence.sha256

@@ -0,0 +1 @@
+690900a416a12668a5b2249081d8dd3b7cbafce1c27faac835f815a0fc5ff4ed  evidence.json