ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter9/prompt-auto-optimization/.gitignore

@@ -0,0 +1,13 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.venv/
+venv/
+
+# 环境变量与密钥
+.env
+
+# 运行时生成的工件（Coding Agent 改写的工作副本、运行日志等）
+runtime/
+output/

chapter9/prompt-auto-optimization/README.md

@@ -0,0 +1,112 @@
+# 实验 9-3：基于失败轨迹优化系统 Prompt
+
+本实验使用航空客服的“过度转接”案例，演示一条受控的 Prompt 学习链路：先评测运行轨迹，再把失败整理为结构化诊断，随后由 Coding Agent 生成最小补丁，最后用边界集与旧任务保留集决定待验证版本是否可以灰度发布。
+
+这与一次性人工提示工程的关键差别，不在于“让模型改写 Prompt”，而在于每个补丁都能回答三个问题：它由哪些失败案例触发、作用于哪条规则、为什么没有破坏旧行为。
+
+## 实验流程
+
+`evaluate.py` 运行保留集与边界集。`learning_signal.py` 将每条轨迹拆成规则遵从、任务解决和合规变通三个维度，并保留来源 case ID。`coding_agent.py` 读取结构化报告，对 Prompt 做精确的 `old_str → new_str` 编辑。`release_gate.py` 生成待验证 manifest，并执行四项发布检查：补丁非空、来源可追溯、保留集不退化、边界集确有改善。
+
+待验证补丁只写入 `runtime/system_prompt_working.txt`，不会覆盖 `prompts/system_prompt.txt`。门槛通过时，实验只返回 `release_to_canary`，表示允许灰度；未通过则返回 `reject_candidate`。
+
+```text
+失败轨迹 → 三维诊断 → 最小 Prompt diff → 待验证 manifest
+                                         ↓
+                         边界集改善 + 保留集不退化
+                                         ↓
+                              灰度发布或拒绝提案
+```
+
+## 运行
+
+完整实验需要一个 OpenAI 兼容的模型接口：
+
+```bash
+# 从仓库根目录开始：使用共享的第 8 章环境
+uv sync --locked --python 3.12 --extra ch8
+# Apple Silicon macOS 需要 macOS 14+（锁文件中的 bitsandbytes wheel 要求）；
+# 更早的 macOS 请使用下方单项目兼容路径。
+
+# 切换目录前先激活环境：
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .\.venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+# 未安装 uv 时可用 pip 兜底：
+# python -m pip install -e ".[ch8]"
+
+cd chapter8/prompt-auto-optimization
+
+# 迁移期间仍支持单项目兼容路径：
+# python -m pip install -r requirements.txt
+
+cp env.example .env
+export OPENAI_API_KEY=your_api_key_here
+python demo.py --quick --model gpt-5.6
+python demo.py --model gpt-5.6 --output output/run.json
+```
+
+以上两条命令会真实调用客服 Agent、LLM Judge 和 Coding Agent，并非 dry-run；`--quick` 只是减少评测案例数量。`python demo.py --dry-run` 仅检查模型配置和用例选择，不生成补丁，也不能作为实验结果。
+
+离线可以检查参数、诊断逻辑和发布门槛：
+
+```bash
+# 在仓库根目录安装包含 pytest 的测试环境：
+uv sync --locked --python 3.12 --extra ch8 --extra dev
+
+# 未安装 uv 时可用 pip 测试环境兜底：
+# python -m pip install -e ".[ch8,dev]"
+
+source .venv/bin/activate
+cd chapter8/prompt-auto-optimization
+
+python demo.py --dry-run
+python -m pytest tests
+```
+
+项目也保留人工调优版 `prompts/system_prompt_manual.txt` 作为对照。完整实验比较初始版、自动提案版和人工版在两组任务上的表现；具体准确率会随被测模型变化，是否发布则始终由显式门槛决定，而不是由 Coding Agent 自己决定。
+
+### 正文验收运行（2026-07-30）
+
+正文的正式入口会强制使用完整的 5 条保留任务和 5 条边界任务，不接受 `--quick` 作为验收：
+
+```bash
+python run_experiment_9_3.py \
+  --provider ark \
+  --model doubao-seed-1-6-flash-250615 \
+  --rounds 3
+```
+
+机器可读证据位于 `validation/real_20260729T171101Z/evidence.json`，SHA-256 为
+`491b54ca5e10ea3b3154c014a44039e9520ae61880e4d46e7f667fa0aa2c4106`；
+`validation/latest.json` 指向同一内容。证据保存了 73 次无凭据原始 API 请求/响应、三份 Prompt 的逐例轨迹、Judge 理由、精确 `old_str → new_str` 编辑、来源 case ID、待验证 manifest、发布检查、Token 用量和耗时。
+
+本次真实结果如下：
+
+| Prompt | 保留集 | 过度转接边界集 |
+| --- | ---: | ---: |
+| 初始 Prompt | 5/5 | 0/5 |
+| 自动提案 | 5/5 | 2/5 |
+| 人工一次性调优 | 5/5 | 4/5 |
+
+自动提案满足“补丁非空且可审计、来源可追溯、边界集改善、保留集不退化”，因此结果是
+`release_to_canary`，不是覆盖稳定 Prompt 或直接全量发布。自动提案虽通过正文门槛，但仍明显弱于人工对照；证据没有把 2/5 描述成边界问题已全部解决。
+
+ARK 回执合计 73,456 个输入 Token、7,313 个输出 Token、80,769 个 Token。该接口没有返回货币费用字段，所以证据中的美元成本保持 `null`，没有用未固定的价目表猜算。
+
+## 文件说明
+
+| 文件 | 作用 |
+| --- | --- |
+| `airline_env.py` | 工具调用环境与保留/边界案例 |
+| `evaluate.py` | 运行 Agent，输出轨迹结果与处理判定 |
+| `learning_signal.py` | 从失败轨迹生成三维诊断和来源证据 |
+| `coding_agent.py` | 生成并应用可审计的最小 Prompt 编辑 |
+| `release_gate.py` | 待验证 manifest、回归门槛和发布决定 |
+| `demo.py` | 串联完整闭环并输出对照结果 |
+| `tests/` | 离线验证诊断、补丁应用、工具空值处理、接受和拒绝路径 |
+| `run_experiment_9_3.py` | 强制完整三组真实验收并保存原始回执与 `acceptance` |
+
+本实验使用无外部副作用的航空客服沙盒，以便三份 Prompt 在完全相同的状态和任务上重复执行。它完成了正文规定的实验对照，但不等同于生产航空系统验收；接入生产时仍须把规则遵从连接到正式政策与订单真值，并扩充专家校准和安全留出集。

chapter9/prompt-auto-optimization/airline_env.py

@@ -0,0 +1,327 @@
+"""
+精简版「航空客服」模拟环境（对标 tau-bench 的航空场景，但去掉复杂度）。
+
+包含三部分：
+1. TOOLS      —— 暴露给 Agent 的工具（含关键的 transfer_to_human）。
+2. run_agent  —— 一个带工具调用循环的最小 Agent：给定 system prompt 和用户请求，
+                 返回它是否转接人工、以及最终回复。
+3. CASES      —— 两组评测用例：
+                 - 保留任务集(holdout)：正常请求，Agent 应正确处理（不该转的别转，该转的要转）。
+                 - 边界案例集(boundary)：政策争议，Agent 应解释政策而非一转了之。
+"""
+
+import json
+from config import get_client, get_model, get_temperature, record_completion
+
+# ----------------------------------------------------------------------------
+# 1. 工具定义（OpenAI function-calling 格式）
+# ----------------------------------------------------------------------------
+TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "lookup_reservation",
+            "description": "根据订单号查询乘客的订单详情（航班、舱位、票价类型等）。",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "confirmation_code": {"type": "string", "description": "订单号"}
+                },
+                "required": ["confirmation_code"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "change_flight",
+            "description": "为乘客办理改签到指定的新航班。",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "confirmation_code": {"type": "string"},
+                    "new_flight": {"type": "string", "description": "新航班号或日期"},
+                },
+                "required": ["confirmation_code", "new_flight"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "get_refund_policy",
+            "description": "查询退票/退款政策。传入票价类型（如 经济舱特价票/全价经济舱/商务舱）。",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "fare_type": {"type": "string", "description": "票价类型"}
+                },
+                "required": ["fare_type"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "get_baggage_policy",
+            "description": "查询行李额与逾重费政策。传入舱位等级。",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "cabin": {"type": "string", "description": "舱位等级，如 经济舱/商务舱"}
+                },
+                "required": ["cabin"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "change_seat",
+            "description": "为乘客办理选座或换座。",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "confirmation_code": {"type": "string"},
+                    "seat": {"type": "string", "description": "目标座位号"},
+                },
+                "required": ["confirmation_code", "seat"],
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "transfer_to_human",
+            "description": "把对话转接给人工客服。调用后 Agent 不再继续处理本次请求。",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "reason": {"type": "string", "description": "转接原因"}
+                },
+                "required": ["reason"],
+            },
+        },
+    },
+]
+
+# ----------------------------------------------------------------------------
+# 2. 工具的 mock 实现（返回固定的模拟数据，供 Agent 组织回复）
+# ----------------------------------------------------------------------------
+_POLICY_REFUND = {
+    "经济舱特价票": "经济舱特价票为不可退票产品，不支持自愿退款；如未起飞可申请退还机建燃油等税费。",
+    "全价经济舱": "全价经济舱起飞前可退，收取 5% 退票手续费。",
+    "商务舱": "商务舱起飞前可全额退票，不收手续费。",
+}
+
+
+def _run_tool(name: str, args: dict) -> str:
+    """执行工具，返回给模型的字符串结果。"""
+    if name == "lookup_reservation":
+        return json.dumps(
+            {
+                "confirmation_code": args.get("confirmation_code", "UNKNOWN"),
+                "passenger": "张伟",
+                "flight": "YS1234 上海虹桥→北京首都 2026-08-01 09:00",
+                "cabin": "经济舱",
+                "fare_type": "经济舱特价票",
+                "status": "已出票",
+            },
+            ensure_ascii=False,
+        )
+    if name == "change_flight":
+        return json.dumps(
+            {"result": "success", "new_flight": args.get("new_flight"), "fee": "改签费 200 元"},
+            ensure_ascii=False,
+        )
+    if name == "get_refund_policy":
+        fare = args.get("fare_type", "经济舱特价票")
+        text = _POLICY_REFUND.get(fare, _POLICY_REFUND["经济舱特价票"])
+        return json.dumps({"fare_type": fare, "policy": text}, ensure_ascii=False)
+    if name == "get_baggage_policy":
+        cabin = args.get("cabin") or "经济舱"
+        free = "20kg" if "经济" in cabin else "30kg"
+        return json.dumps(
+            {"cabin": cabin, "free_allowance": free, "excess_fee": "逾重费 50 元/kg"},
+            ensure_ascii=False,
+        )
+    if name == "change_seat":
+        return json.dumps(
+            {"result": "success", "seat": args.get("seat")}, ensure_ascii=False
+        )
+    return json.dumps({"result": "ok"}, ensure_ascii=False)
+
+
+# ----------------------------------------------------------------------------
+# 3. 最小 Agent 循环
+# ----------------------------------------------------------------------------
+def run_agent(system_prompt: str, user_message: str, max_steps: int = 4) -> dict:
+    """
+    运行一次客服会话。返回：
+      {
+        "transferred": bool,        # 是否调用了 transfer_to_human
+        "transfer_reason": str|None,
+        "final_text": str,          # Agent 面向乘客的最终回复（若转接则为空）
+        "tool_calls": [str, ...],   # 依次调用过的工具名
+      }
+    """
+    client = get_client()
+    model = get_model()
+    messages = [
+        {"role": "system", "content": system_prompt},
+        {"role": "user", "content": user_message},
+    ]
+    tool_calls_log = []
+
+    for _ in range(max_steps):
+        request = dict(
+            model=model,
+            messages=messages,
+            tools=TOOLS,
+            temperature=get_temperature(),
+        )
+        resp = record_completion(client, kind="task_agent", **request)
+        msg = resp.choices[0].message
+
+        if not msg.tool_calls:
+            # 模型直接给出面向乘客的文字回复 —— 会话结束
+            return {
+                "transferred": False,
+                "transfer_reason": None,
+                "final_text": msg.content or "",
+                "tool_calls": tool_calls_log,
+            }
+
+        # 有工具调用，先把 assistant 消息加入历史
+        messages.append(msg)
+
+        transferred_now = False
+        transfer_reason = None
+        for tc in msg.tool_calls:
+            name = tc.function.name
+            try:
+                args = json.loads(tc.function.arguments or "{}")
+            except json.JSONDecodeError:
+                args = {}
+            tool_calls_log.append(name)
+
+            if name == "transfer_to_human":
+                transferred_now = True
+                transfer_reason = args.get("reason", "")
+                result = json.dumps({"result": "transferred"}, ensure_ascii=False)
+            else:
+                result = _run_tool(name, args)
+
+            messages.append(
+                {"role": "tool", "tool_call_id": tc.id, "content": result}
+            )
+
+        if transferred_now:
+            return {
+                "transferred": True,
+                "transfer_reason": transfer_reason,
+                "final_text": "",
+                "tool_calls": tool_calls_log,
+            }
+
+    # 超出步数上限仍未收敛，兜底再要一次纯文本回复
+    resp = record_completion(
+        client, kind="task_agent_forced_final",
+        model=model, messages=messages, temperature=get_temperature(),
+    )
+    return {
+        "transferred": False,
+        "transfer_reason": None,
+        "final_text": resp.choices[0].message.content or "",
+        "tool_calls": tool_calls_log,
+    }
+
+
+# ----------------------------------------------------------------------------
+# 4. 评测用例
+# ----------------------------------------------------------------------------
+# 每个用例字段：
+#   id / group / user   —— 用户请求
+#   should_transfer     —— 正确行为是否应当转接人工
+#   rubric              —— 当 should_transfer=False 时，判定"是否妥善处理"的要点
+CASES = [
+    # ---------- 保留任务集(holdout)：既有正确行为，优化后不能退化 ----------
+    {
+        "id": "H1-改签",
+        "group": "holdout",
+        "user": "你好，我的订单号是 ABC123，想把 8 月 1 日的航班改到 8 月 3 日，帮我改一下。",
+        "should_transfer": False,
+        "rubric": "查询订单并为乘客办理/说明改签到 8 月 3 日的相关信息（如改签费），而不是转人工。",
+    },
+    {
+        "id": "H2-行李额",
+        "group": "holdout",
+        "user": "我买的是经济舱，免费托运行李额是多少？超重了怎么收费？",
+        "should_transfer": False,
+        "rubric": "给出经济舱免费行李额与逾重费的具体政策信息，而不是转人工。",
+    },
+    {
+        "id": "H3-选座",
+        "group": "holdout",
+        "user": "订单号 ABC123，我想换到靠窗的 32A 座位。",
+        "should_transfer": False,
+        "rubric": "为乘客办理/确认换到 32A 座位，而不是转人工。",
+    },
+    {
+        "id": "H4-明确要人工",
+        "group": "holdout",
+        "user": "别跟我扯了，我就要跟人工客服说话，马上给我转人工。",
+        "should_transfer": True,  # 乘客明确要求人工 —— 这类本就该转，优化后仍要能转
+        "rubric": "乘客明确要求人工客服，应当转接。",
+    },
+    {
+        "id": "H5-紧急安全",
+        "group": "holdout",
+        "user": "我现在在机场，同行的老人突然晕倒了，赶不上 YS1234 航班了，需要马上有人帮忙处理！",
+        "should_transfer": True,  # 紧急安全 —— 本就该转，优化后仍要能转
+        "rubric": "涉及紧急人身安全，应当转接人工立即介入。",
+    },
+
+    # ---------- 边界案例集(boundary)：政策争议，应解释政策而非一转了之 ----------
+    {
+        "id": "B1-不可退票要退款",
+        "group": "boundary",
+        "user": "我要求全额退款！订单 ABC123。我不管，我就要全退。",
+        "should_transfer": False,
+        "rubric": "查询退票政策后，耐心解释该票（经济舱特价票）为不可退票产品、可退税费，并给出可行替代方案，而不是转人工。",
+    },
+    {
+        "id": "B2-要求免改签费",
+        "group": "boundary",
+        "user": "改签还要收我 200 块？凭什么！你必须给我免掉这个改签费，不然没完。",
+        "should_transfer": False,
+        "rubric": "耐心解释改签费的政策依据，说明无法违规减免，并提供合规的替代方案，而不是因为乘客不满就转人工。",
+    },
+    {
+        "id": "B3-小延误索赔",
+        "group": "boundary",
+        "user": "航班晚了半小时，害我差点误事，我要求航空公司赔偿我 500 块！",
+        "should_transfer": False,
+        "rubric": "共情并解释延误补偿的政策门槛（30 分钟的短延误通常不达补偿标准），说明处理口径，而不是转人工。",
+    },
+    {
+        "id": "B4-索要免费升舱",
+        "group": "boundary",
+        "user": "我是老客户了，这次必须给我免费升到商务舱，这点面子都不给？",
+        "should_transfer": False,
+        "rubric": "礼貌解释免费升舱不符合政策、说明可付费升舱或积分兑换等合规途径，而不是转人工。",
+    },
+    {
+        "id": "B5-超额免费行李",
+        "group": "boundary",
+        "user": "我经济舱，这次要带 3 件行李，你们必须都给我免费托运，别收钱。",
+        "should_transfer": False,
+        "rubric": "解释经济舱的免费行李额与超出部分的收费政策，说明无法全部免费，并给出合规建议，而不是转人工。",
+    },
+]
+
+
+def get_cases(group: str = None):
+    if group is None:
+        return CASES
+    return [c for c in CASES if c["group"] == group]

chapter9/prompt-auto-optimization/coding_agent.py

@@ -0,0 +1,189 @@
+"""
+Coding Agent：读取系统提示词文件 → 定位相关规则 → 生成精确的搜索/替换编辑 → 真的改写文件。
+
+它的工作方式和真实的编程 Agent（如 Claude Code / Cursor）一致：
+不是让模型整篇重写，而是让模型产出一组 (old_str -> new_str) 的精确编辑，
+由代码逐条做"精确字符串替换"落到文件里；若某条编辑的 old_str 匹配不上，
+把错误反馈回模型让它重试。这样修改是"代码级"的、可审计的（能直接出 diff）。
+"""
+
+import difflib
+import json
+from config import get_client, get_model, get_temperature, record_completion
+
+# 暴露给 Coding Agent 的"文件编辑工具"
+EDIT_TOOLS = [
+    {
+        "type": "function",
+        "function": {
+            "name": "apply_edits",
+            "description": (
+                "对提示词文件应用一组精确的搜索/替换编辑。每条编辑给出 old_str "
+                "（文件中唯一存在的原文片段）和 new_str（替换后的新文本）。"
+                "old_str 必须与文件内容逐字符完全一致。"
+            ),
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "edits": {
+                        "type": "array",
+                        "items": {
+                            "type": "object",
+                            "properties": {
+                                "old_str": {"type": "string"},
+                                "new_str": {"type": "string"},
+                            },
+                            "required": ["old_str", "new_str"],
+                        },
+                    },
+                    "rationale": {
+                        "type": "string",
+                        "description": "简述本次改动如何回应人类反馈。",
+                    },
+                },
+                "required": ["edits"],
+            },
+        },
+    }
+]
+
+
+def _apply_one(content: str, old_str: str, new_str: str) -> tuple[str, str | None]:
+    """尝试应用一条编辑。成功返回(新内容, None)，失败返回(原内容, 错误信息)。"""
+    if old_str is None or new_str is None:
+        return content, "old_str/new_str 不能为 null"
+    count = content.count(old_str)
+    if count == 0:
+        return content, f"old_str 在文件中未找到：{old_str[:60]!r}"
+    if count > 1:
+        return content, f"old_str 在文件中出现 {count} 次（不唯一）：{old_str[:60]!r}"
+    return content.replace(old_str, new_str, 1), None
+
+
+def _apply_edits_from_args(working: str, args: dict) -> tuple[str, int, list, list, list]:
+    """Apply edits; null edits → []; skip non-dict entries with a warning."""
+    edits = args.get("edits")
+    if edits is None:
+        edits = []
+    errors = []
+    warnings = []
+    applied = 0
+    for e in edits:
+        if not isinstance(e, dict):
+            warnings.append(f"跳过非对象编辑项 ({type(e).__name__}): {e!r}")
+            continue
+        working, err = _apply_one(working, e.get("old_str", ""), e.get("new_str", ""))
+        if err:
+            errors.append(err)
+        else:
+            applied += 1
+    return working, applied, errors, warnings, edits
+
+def optimize_prompt(prompt_path: str, feedback, max_rounds: int = 3, verbose: bool = True) -> dict:
+    """
+    让 Coding Agent 根据 human feedback 改写 prompt_path 指向的文件（原地覆盖）。
+
+    返回 {"before": 原文, "after": 新文, "diff": 统一 diff 文本, "rationale": 说明}。
+    """
+    client = get_client()
+    model = get_model()
+
+    with open(prompt_path, "r", encoding="utf-8") as f:
+        original = f.read()
+
+    feedback_text = json.dumps(feedback, ensure_ascii=False, indent=2) if isinstance(feedback, dict) else str(feedback)
+
+    system = (
+        "你是一名资深的提示词工程 Coding Agent。你会收到一份航空客服 Agent 的"
+        "系统提示词文件，以及从失败轨迹生成的结构化诊断。请定位与'人工转接'相关的规则，"
+        "生成精确的搜索/替换编辑来改进它，然后调用 apply_edits 工具落地修改。\n"
+        "改动目标：\n"
+        "1) 把转接的边界收紧、明确为仅两种情况：乘客明确要求人工客服、以及紧急安全情况；\n"
+        "2) 删除或改写会诱发'过度转接'的模糊规则（如'不确定或乘客不满就转接'）；\n"
+        "3) 新增一条明确的负面规则：绝不因政策争议 / 乘客不满而转接，而应先查政策、"
+        "耐心解释并提供合规的替代方案。\n"
+        "只修改与转接策略相关的部分，尽量保留其余内容不动。"
+    )
+
+    messages = [
+        {"role": "system", "content": system},
+        {
+            "role": "user",
+            "content": (
+                f"【失败轨迹诊断】\n{feedback_text}\n\n"
+                f"【当前系统提示词文件内容】\n---\n{original}\n---\n\n"
+                "请调用 apply_edits 提交你的精确编辑。"
+            ),
+        },
+    ]
+
+    working = original
+    rationale = ""
+    submitted_edits = []
+
+    for round_idx in range(max_rounds):
+        resp = record_completion(client, kind="coding_agent",
+            model=model,
+            messages=messages,
+            tools=EDIT_TOOLS,
+            tool_choice={"type": "function", "function": {"name": "apply_edits"}},
+            temperature=get_temperature(),
+        )
+        msg = resp.choices[0].message
+        messages.append(msg)
+
+        if not msg.tool_calls:
+            break
+
+        # 处理（唯一的）apply_edits 调用
+        tc = msg.tool_calls[0]
+        try:
+            args = json.loads(tc.function.arguments or "{}")
+        except json.JSONDecodeError:
+            args = {}
+        rationale = args.get("rationale", rationale)
+        working, applied, errors, warnings, edits = _apply_edits_from_args(working, args)
+        submitted_edits = edits
+
+        if verbose:
+            print(f"  [round {round_idx + 1}] 提交 {len(edits)} 条编辑，成功 {applied}，失败 {len(errors)}，跳过 {len(warnings)}")
+
+        if not errors:
+            # 有效编辑已全部成功应用，落盘
+            msg_content = "所有有效编辑已成功应用。"
+            if warnings:
+                msg_content += "\n以下非对象编辑项已被跳过：\n" + "\n".join(f"- {w}" for w in warnings)
+            messages.append(
+                {"role": "tool", "tool_call_id": tc.id, "content": msg_content}
+            )
+            break
+        else:
+            # 有实际应用失败：回滚到原文，把错误反馈给模型重试（保持编辑的原子性）
+            working = original
+            feedback_msg = (
+                "以下编辑未能应用，请修正后重新提交完整的编辑列表（注意 old_str 必须与文件逐字符一致）：\n"
+                + "\n".join(f"- {er}" for er in errors)
+            )
+            if warnings:
+                feedback_msg += "\n另有以下非对象编辑项已被跳过：\n" + "\n".join(f"- {w}" for w in warnings)
+            messages.append({"role": "tool", "tool_call_id": tc.id, "content": feedback_msg})
+    # 落盘（原地覆盖 prompt 文件）
+    with open(prompt_path, "w", encoding="utf-8") as f:
+        f.write(working)
+
+    diff = "".join(
+        difflib.unified_diff(
+            original.splitlines(keepends=True),
+            working.splitlines(keepends=True),
+            fromfile="system_prompt.txt (before)",
+            tofile="system_prompt.txt (after)",
+        )
+    )
+
+    return {
+        "before": original,
+        "after": working,
+        "diff": diff,
+        "rationale": rationale,
+        "edits": submitted_edits,
+    }

chapter9/prompt-auto-optimization/config.py

@@ -0,0 +1,223 @@
+"""
+统一的 LLM 客户端配置。
+
+默认使用 OpenAI（读取 OPENAI_API_KEY，模型 gpt-5.6-luna）。
+也支持通过环境变量 LLM_PROVIDER 切换到 Moonshot / 火山方舟(ARK)，
+它们都兼容 OpenAI 的 Chat Completions + 工具调用接口。
+
+    export LLM_PROVIDER=openai   # 默认
+    export LLM_PROVIDER=moonshot # 用 MOONSHOT_API_KEY
+    export LLM_PROVIDER=ark      # 用 ARK_API_KEY，并需设置 ARK_MODEL
+
+统一的 OpenRouter 兜底（fallback）：
+    若所选 provider 自己的 Key 缺失，但设置了 OPENROUTER_API_KEY，则自动改走
+    OpenRouter（https://openrouter.ai/api/v1），并把模型名映射到 OpenRouter 命名：
+        gpt-*     -> openai/gpt-*
+        claude-*  -> anthropic/claude-opus-4.8
+        含 "/"    -> 原样透传
+        其它      -> openai/gpt-5.6-luna
+"""
+
+import os
+import time
+from typing import Any
+from openai import OpenAI
+from dotenv import load_dotenv
+
+load_dotenv()
+
+OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
+
+# 各提供商的默认配置：base_url / 环境变量名 / 默认模型
+_PROVIDERS = {
+    "openai": {
+        "base_url": None,  # 使用 SDK 默认
+        "key_env": "OPENAI_API_KEY",
+        "default_model": "gpt-5.6-luna",
+    },
+    "moonshot": {
+        "base_url": "https://api.moonshot.cn/v1",
+        "key_env": "MOONSHOT_API_KEY",
+        "default_model": "kimi-k3",
+    },
+    "ark": {
+        "base_url": "https://ark.cn-beijing.volces.com/api/v3",
+        "key_env": "ARK_API_KEY",
+        # ARK 需要用推理接入点(endpoint id) 作为 model，请通过 ARK_MODEL 指定
+        "default_model": os.getenv("ARK_MODEL", "doubao-seed-1-6-250615"),
+    },
+    "openrouter": {
+        "base_url": OPENROUTER_BASE_URL,
+        "key_env": "OPENROUTER_API_KEY",
+        "default_model": "openai/gpt-4o-mini",
+    },
+}
+
+API_TURNS = []
+
+
+def _jsonable(value: Any) -> Any:
+    if hasattr(value, "model_dump"):
+        return _jsonable(value.model_dump(mode="json", exclude_none=True))
+    if isinstance(value, dict):
+        return {str(key): _jsonable(item) for key, item in value.items()}
+    if isinstance(value, (list, tuple)):
+        return [_jsonable(item) for item in value]
+    return value
+
+
+def get_provider() -> str:
+    return os.getenv("LLM_PROVIDER", "openai").lower().strip()
+
+
+def _to_openrouter_model(model: str) -> str:
+    """把常见模型名映射到 OpenRouter 命名空间。"""
+    if not model:
+        return "openai/gpt-5.6-luna"
+    if "/" in model:
+        return model
+    if model.startswith("gpt-"):
+        return "openai/" + model
+    if model.startswith("claude-"):
+        return "anthropic/claude-opus-4.8"
+    return "openai/gpt-5.6-luna"
+
+
+def _is_reasoning_model(model: str) -> bool:
+    """gpt-5.x / o1·o3·o4 / kimi-k3 / *reasoner 等推理模型：不接受 temperature=0，
+    直连 gpt-5.x 还需组织实名且工具调用受限，故优先走 OpenRouter。"""
+    m = (model or "").lower()
+    return (m.startswith(("gpt-5", "o1", "o3", "o4"))
+            or m.startswith("kimi-k3")
+            or "reasoner" in m or "thinking" in m)
+
+
+def _use_openrouter(cfg: dict) -> bool:
+    """走 OpenRouter 的两种情形：
+    1) provider 自己的 Key 缺失、但有 OPENROUTER_API_KEY（统一兜底）；
+    2) 目标是 gpt-5.x 且有 OPENROUTER_API_KEY —— 直连 gpt-5.x 需组织实名、
+       且 /chat/completions 工具调用受限，故即便有 OPENAI_API_KEY 也优先 OpenRouter。"""
+    if not os.getenv("OPENROUTER_API_KEY"):
+        return False
+    if not os.getenv(cfg["key_env"]):
+        return True
+    model = os.getenv("LLM_MODEL") or cfg["default_model"]
+    return (model or "").lower().startswith("gpt-5")
+
+
+def get_model() -> str:
+    """允许用 LLM_MODEL 覆盖默认模型；OpenRouter 兜底路径下映射模型名。"""
+    provider = get_provider()
+    if provider not in _PROVIDERS:
+        raise ValueError(f"未知的 LLM_PROVIDER: {provider}")
+    cfg = _PROVIDERS[provider]
+    model = os.getenv("LLM_MODEL") or cfg["default_model"]
+    if _use_openrouter(cfg):
+        return _to_openrouter_model(model)
+    return model
+
+
+def get_client() -> OpenAI:
+    provider = get_provider()
+    if provider not in _PROVIDERS:
+        raise ValueError(f"未知的 LLM_PROVIDER: {provider}")
+    cfg = _PROVIDERS[provider]
+    if _use_openrouter(cfg):
+        return OpenAI(api_key=os.getenv("OPENROUTER_API_KEY"), base_url=OPENROUTER_BASE_URL)
+    api_key = os.getenv(cfg["key_env"])
+    if not api_key:
+        raise RuntimeError(
+            f"环境变量 {cfg['key_env']} 未设置，也未设置 OPENROUTER_API_KEY。"
+            f"请参考 env.example 配置其一（OpenRouter 可作为统一兜底）后重试。"
+        )
+    kwargs = {"api_key": api_key}
+    if cfg["base_url"]:
+        kwargs["base_url"] = cfg["base_url"]
+    return OpenAI(**kwargs)
+
+
+def record_completion(client: OpenAI, *, kind: str, **request: Any):
+    """Execute and retain a credential-free raw request/response receipt."""
+    started = time.time()
+    response = client.chat.completions.create(**request)
+    API_TURNS.append({
+        "kind": kind,
+        "provider": get_provider(),
+        "endpoint": get_backend_metadata()["endpoint"],
+        "request": _jsonable(request),
+        "response": response.model_dump(mode="json", exclude_none=True),
+        "elapsed_seconds": round(time.time() - started, 6),
+    })
+    return response
+
+
+def reset_api_turns() -> None:
+    API_TURNS.clear()
+
+
+def get_api_turns() -> list[dict]:
+    return list(API_TURNS)
+
+
+def get_backend_metadata() -> dict[str, Any]:
+    provider = get_provider()
+    cfg = _PROVIDERS[provider]
+    if _use_openrouter(cfg):
+        base_url = OPENROUTER_BASE_URL
+        key_env = "OPENROUTER_API_KEY"
+        routed_provider = "openrouter"
+    else:
+        base_url = cfg["base_url"] or "https://api.openai.com/v1"
+        key_env = cfg["key_env"]
+        routed_provider = provider
+    return {
+        "configured_provider": provider,
+        "routed_provider": routed_provider,
+        "model": get_model(),
+        "endpoint": f"{base_url}/chat/completions",
+        "credential_source_env": key_env,
+        "credential_value_recorded": False,
+    }
+
+
+def usage_summary() -> dict[str, Any]:
+    prompt = completion = total = 0
+    native_cost = 0.0
+    native_cost_count = 0
+    for turn in API_TURNS:
+        usage = turn.get("response", {}).get("usage") or {}
+        prompt += int(usage.get("prompt_tokens") or 0)
+        completion += int(usage.get("completion_tokens") or 0)
+        total += int(usage.get("total_tokens") or 0)
+        if usage.get("cost") is not None:
+            native_cost += float(usage["cost"])
+            native_cost_count += 1
+    return {
+        "prompt_tokens": prompt,
+        "completion_tokens": completion,
+        "total_tokens": total or prompt + completion,
+        "provider_reported_cost_usd": round(native_cost, 9) if native_cost_count else None,
+        "provider_reported_cost_observations": native_cost_count,
+        "cost_qualification": (
+            "provider-native usage.cost summed across calls"
+            if native_cost_count else "provider did not expose monetary cost; no price was guessed"
+        ),
+    }
+
+
+# 全部 LLM 调用统一使用低温度，保证结果可复现；
+# 但推理模型（gpt-5.x / o 系列 / kimi-k3 等）只接受默认 temperature=1，
+# 故按当前解析出的模型自动选择默认温度（可用 LLM_TEMPERATURE 显式覆盖）。
+def _default_temperature() -> str:
+    provider = get_provider()
+    cfg = _PROVIDERS.get(provider, _PROVIDERS["openai"])
+    model = os.getenv("LLM_MODEL") or cfg["default_model"]
+    return "1" if _is_reasoning_model(model) else "0"
+
+
+def get_temperature() -> float:
+    """在调用时按当前解析出的模型选择温度，使 CLI/env 的 --model/--provider
+    覆盖生效。原来的模块级 TEMPERATURE 常量在 import 时就被固定，而 demo.py 在
+    import 之后才设置 LLM_MODEL/LLM_PROVIDER，导致温度停留在默认模型的值
+    （例如把非推理模型误用 temperature=1，破坏了本文件追求的可复现性）。"""
+    return float(os.getenv("LLM_TEMPERATURE", _default_temperature()))

chapter9/prompt-auto-optimization/demo.py

@@ -0,0 +1,330 @@
+"""
+实验 9-3：基于失败轨迹的系统提示词自动优化
+
+一条命令跑通完整流程：
+  1. 用【初始 prompt】评测 → 暴露"政策争议就转人工"的过度转接问题；
+  2. 从失败轨迹生成三维诊断，保留来源案例；
+  3. Coding Agent 生成候选 prompt 的最小 diff；
+  4. 用边界集与保留集决定候选版本是否可灰度发布；
+  5. 与人工调优版对照。
+
+    python demo.py            # 完整运行：10 个用例 × 3 份 prompt
+    python demo.py --quick    # 快速演示：每组只取 2 个用例，省时省钱
+    python demo.py --help     # 查看全部命令行参数（中文说明）
+"""
+
+import argparse
+import json
+import os
+import shutil
+import sys
+import time
+from datetime import datetime, timezone
+
+from evaluate import evaluate_prompt
+from coding_agent import optimize_prompt
+from config import (
+    get_api_turns,
+    get_backend_metadata,
+    get_provider,
+    get_model,
+    reset_api_turns,
+    usage_summary,
+)
+from airline_env import CASES
+from learning_signal import diagnose_failures, format_learning_signal
+from release_gate import build_candidate_manifest, evaluate_release_gate
+
+GROUPS = ("holdout", "boundary")
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+INITIAL_PROMPT = os.path.join(HERE, "prompts", "system_prompt.txt")
+MANUAL_PROMPT = os.path.join(HERE, "prompts", "system_prompt_manual.txt")
+WORKING_PROMPT = os.path.join(HERE, "runtime", "system_prompt_working.txt")
+
+def _read(path):
+    with open(path, "r", encoding="utf-8") as f:
+        return f.read()
+
+
+def _pct(cn):
+    c, n = cn
+    return f"{c}/{n} ({100 * c / n:.0f}%)" if n else "-"
+
+
+def print_table(rows):
+    """rows: list of (label, holdout_tuple, boundary_tuple)"""
+    print("\n" + "=" * 74)
+    print("正确率对比（保留任务集 = 既有正确行为不能退化；边界案例集 = 过度转接应改善）")
+    print("=" * 74)
+    header = f"{'系统提示词版本':<26}{'保留任务集(holdout)':<20}{'边界案例集(boundary)':<20}"
+    print(header)
+    print("-" * 74)
+    for label, holdout, boundary in rows:
+        print(f"{label:<24}{_pct(holdout):<22}{_pct(boundary):<22}")
+    print("=" * 74)
+
+
+def _select_cases(limit_per_group=None, groups=GROUPS):
+    """按分组筛选用例，并对每组最多取 limit_per_group 个（None 表示不限制）。"""
+    picked, counts = [], {}
+    for c in CASES:
+        g = c["group"]
+        if g not in groups:
+            continue
+        if limit_per_group and counts.get(g, 0) >= limit_per_group:
+            continue
+        picked.append(c)
+        counts[g] = counts.get(g, 0) + 1
+    return picked
+
+
+def main(cases=None, rounds=3, output=None):
+    if cases is None:
+        cases = CASES
+    reset_api_turns()
+    campaign_started = time.time()
+    print("#" * 74)
+    print("# 实验 9-3：基于失败轨迹的系统提示词自动优化（航空客服场景）")
+    print(f"# LLM 提供商: {get_provider()}   模型: {get_model()}")
+    print(f"# 用例数: {len(cases)}（保留集 + 边界集）   Coding Agent 优化轮数上限: {rounds}")
+    print("#" * 74)
+
+    # ---- 准备：把初始 prompt 复制成本次运行的工作副本（Coding Agent 会改写它）----
+    os.makedirs(os.path.dirname(WORKING_PROMPT), exist_ok=True)
+    shutil.copyfile(INITIAL_PROMPT, WORKING_PROMPT)
+
+    # ---- 步骤 1：评测初始 prompt ----
+    print("\n【步骤 1】用初始系统提示词评测（观察是否过度转接）")
+    before = evaluate_prompt(_read(INITIAL_PROMPT), label="初始 prompt", cases=cases)
+    print(
+        f"\n  初始结果：保留集 {_pct(before['holdout'])}，"
+        f"边界集 {_pct(before['boundary'])}"
+    )
+    over_transfer = [
+        r for r in before["results"]
+        if r["group"] == "boundary" and not r["should_transfer"] and r["transferred"]
+    ]
+    print(f"  边界案例中出现【过度转接】的用例数：{len(over_transfer)} / "
+          f"{len([r for r in before['results'] if r['group'] == 'boundary'])}")
+    for r in over_transfer:
+        print(f"    - {r['id']}：政策争议却直接转人工，原因『{r['transfer_reason']}』")
+
+    # ---- 步骤 2：由失败轨迹形成学习信号 ----
+    learning_signal = diagnose_failures(before)
+    print("\n【步骤 2】将失败轨迹整理为三维诊断")
+    print(format_learning_signal(learning_signal))
+
+    # ---- 步骤 3：Coding Agent 生成候选 prompt ----
+    print("\n【步骤 3】Coding Agent 读取诊断并生成候选系统提示词……")
+    candidate_started = time.time()
+    opt = optimize_prompt(WORKING_PROMPT, learning_signal, max_rounds=rounds, verbose=True)
+    failure_to_candidate_seconds = time.time() - candidate_started
+    manifest = build_candidate_manifest(opt, learning_signal)
+    print(f"\n  Coding Agent 改动说明：{opt['rationale']}")
+    print("\n  ---------- 系统提示词文件 diff（真实写入磁盘）----------")
+    print(opt["diff"] if opt["diff"].strip() else "  (无改动)")
+    print("  --------------------------------------------------------")
+
+    print(f"  候选补丁来源：{', '.join(manifest['source_case_ids'])}")
+    print(f"  候选补丁作用域：{manifest['scope']}")
+
+    # ---- 步骤 4：评测候选 prompt 并运行发布门槛 ----
+    print("\n【步骤 4】评测候选系统提示词并运行发布门槛")
+    after = evaluate_prompt(opt["after"], label="自动优化后 prompt", cases=cases)
+    gate = evaluate_release_gate(before, after, manifest)
+    print(f"  发布决定：{gate['decision']}")
+    for check, passed in gate["checks"].items():
+        print(f"    {'✓' if passed else '✗'} {check}")
+
+    # ---- 步骤 5：对照人工调优版 ----
+    print("\n【步骤 5】对照组：人工调优版系统提示词")
+    manual = evaluate_prompt(_read(MANUAL_PROMPT), label="人工调优版 prompt(对照)", cases=cases)
+
+    # ---- 步骤 6：对比表 ----
+    print_table([
+        ("初始 prompt(优化前)", before["holdout"], before["boundary"]),
+        ("自动优化后 prompt", after["holdout"], after["boundary"]),
+        ("人工调优版(对照)", manual["holdout"], manual["boundary"]),
+    ])
+
+    # ---- 结论 ----
+    b_before_c, b_before_n = before["boundary"]
+    b_after_c, _ = after["boundary"]
+    h_before_c, _ = before["holdout"]
+    h_after_c, _ = after["holdout"]
+    print("\n【结论】")
+    print(f"  · 边界案例集正确率：{b_before_c}/{b_before_n} → {b_after_c}/{b_before_n} "
+          f"（{'提升 ✓' if b_after_c > b_before_c else '未提升'}）")
+    print(f"  · 保留任务集正确率：{h_before_c} → {h_after_c} "
+          f"（{'未退化 ✓' if h_after_c >= h_before_c else '退化 ✗'}）")
+    print(f"\n  候选工作副本已写入：{WORKING_PROMPT}")
+    print("  它不会覆盖稳定版本；只有 release_to_canary 才允许进入灰度。")
+
+    # ---- 可选：把对比结果落盘为 JSON，便于复现与二次分析 ----
+    before_by_id = {row["id"]: row for row in before["results"]}
+    after_by_id = {row["id"]: row for row in after["results"]}
+    regressions = [
+        identifier for identifier, old in before_by_id.items()
+        if old["correct"] and not after_by_id[identifier]["correct"]
+    ]
+    boundary_fixed = [
+        identifier for identifier, old in before_by_id.items()
+        if old["group"] == "boundary" and not old["correct"] and after_by_id[identifier]["correct"]
+    ]
+    api_turns = get_api_turns()
+    gates = [
+        {"name": "full_holdout_and_boundary_sets_run", "passed": len(cases) == len(CASES) and {c["group"] for c in cases} == {"holdout", "boundary"}, "evidence": {"selected": len(cases), "canonical": len(CASES)}},
+        {"name": "same_model_and_same_cases_for_three_controls", "passed": all({r["id"] for r in report["results"]} == {c["id"] for c in cases} for report in (before, after, manual)), "evidence": get_model()},
+        {"name": "real_task_agent_calls", "passed": any(turn["kind"].startswith("task_agent") for turn in api_turns), "evidence": sum(turn["kind"].startswith("task_agent") for turn in api_turns)},
+        {"name": "real_llm_judge_calls", "passed": any(turn["kind"] == "llm_judge" for turn in api_turns), "evidence": sum(turn["kind"] == "llm_judge" for turn in api_turns)},
+        {"name": "real_coding_agent_call", "passed": any(turn["kind"] == "coding_agent" for turn in api_turns), "evidence": sum(turn["kind"] == "coding_agent" for turn in api_turns)},
+        {"name": "learning_signal_has_three_dimensions_and_source_ids", "passed": set(learning_signal["dimensions"]) == {"rule_compliance", "task_resolution", "compliant_flexibility"} and bool(learning_signal["source_case_ids"]), "evidence": learning_signal["source_case_ids"]},
+        {"name": "minimal_old_to_new_patch_is_auditable", "passed": bool(manifest.get("edits")) and bool(manifest.get("diff")), "evidence": manifest.get("edits")},
+        {"name": "release_gate_evaluated_all_four_manuscript_conditions", "passed": set(gate["checks"]) >= {"patch_is_nonempty", "patch_is_auditable_old_to_new_edit", "source_cases_are_recorded", "holdout_did_not_regress", "boundary_improved"}, "evidence": gate["checks"]},
+        {"name": "stable_prompt_not_overwritten", "passed": _read(INITIAL_PROMPT) == opt["before"], "evidence": {"stable": INITIAL_PROMPT, "candidate": WORKING_PROMPT}},
+        {"name": "raw_credential_free_api_receipts_saved", "passed": bool(api_turns), "evidence": len(api_turns)},
+    ]
+    execution_accepted = all(item["passed"] for item in gates)
+    result_claims = {
+        "boundary_improved": after["boundary"][0] > before["boundary"][0],
+        "holdout_not_degraded": after["holdout"][0] >= before["holdout"][0],
+        "automatic_candidate_released_only_to_canary": gate["decision"] == "release_to_canary",
+        "automatic_candidate_compared_with_manual": True,
+    }
+    summary = {
+            "schema_version": 2,
+            "experiment_id": "9-3",
+            "canonical_source": "book/chapter9.md#实验-9-3-基于失败轨迹优化系统提示词",
+            "evidence_mode": "real_task_agent_llm_judge_coding_agent_full_campaign",
+            "created_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
+            "provider": get_provider(),
+            "model": get_model(),
+            "backend": get_backend_metadata(),
+            "credential_value_recorded": False,
+            "rounds": rounds,
+            "num_cases": len(cases),
+            "case_ids": [case["id"] for case in cases],
+            "learning_signal": learning_signal,
+            "candidate_manifest": manifest,
+            "release_gate": gate,
+            "rationale": opt["rationale"],
+            "diff": opt["diff"],
+            "prompt_metrics": {
+                "initial_characters": len(opt["before"]),
+                "candidate_characters": len(opt["after"]),
+                "growth_characters": len(opt["after"]) - len(opt["before"]),
+                "manual_characters": len(_read(MANUAL_PROMPT)),
+                "introduced_regressions": len(regressions),
+                "regression_case_ids": regressions,
+                "boundary_failures_fixed": len(boundary_fixed),
+                "boundary_fixed_case_ids": boundary_fixed,
+                "failure_to_candidate_seconds": round(failure_to_candidate_seconds, 6),
+                "campaign_elapsed_seconds": round(time.time() - campaign_started, 6),
+            },
+            "evaluations": {"initial": before, "automatic_candidate": after, "manual": manual},
+            "rows": [
+                {"label": "初始 prompt(优化前)", "holdout": list(before["holdout"]),
+                 "boundary": list(before["boundary"])},
+                {"label": "自动优化后 prompt", "holdout": list(after["holdout"]),
+                 "boundary": list(after["boundary"])},
+                {"label": "人工调优版(对照)", "holdout": list(manual["holdout"]),
+                 "boundary": list(manual["boundary"])},
+            ],
+            "usage": usage_summary(),
+            "api_turns": api_turns,
+            "acceptance": {
+                "gates": gates,
+                "execution_accepted": execution_accepted,
+                "result_claims": result_claims,
+                "all_manuscript_result_claims_observed": all(result_claims.values()),
+            },
+        }
+    if output:
+        os.makedirs(os.path.dirname(os.path.abspath(output)), exist_ok=True)
+        with open(output, "w", encoding="utf-8") as f:
+            json.dump(summary, f, ensure_ascii=False, indent=2)
+        print(f"  对比结果已写入：{output}")
+    return summary
+
+
+def _build_parser():
+    parser = argparse.ArgumentParser(
+        prog="demo.py",
+        description="实验 9-3：从失败轨迹诊断到候选补丁与发布门槛（航空客服场景）。",
+        formatter_class=argparse.RawTextHelpFormatter,
+        epilog=(
+            "示例：\n"
+            "  python demo.py                     # 完整运行：10 个用例 × 3 份 prompt\n"
+            "  python demo.py --quick             # 每组只取 2 个用例，省时省钱\n"
+            "  python demo.py --group boundary    # 只评测边界案例集\n"
+            "  python demo.py --rounds 5 --model gpt-5.6-luna\n"
+            "  python demo.py --output output/run.json  # 把对比结果写成 JSON\n"
+            "  python demo.py --dry-run           # 离线：只打印配置与用例数，不调用 API"
+        ),
+    )
+    parser.add_argument(
+        "--quick", action="store_true",
+        help="快速演示模式：每组只取 2 个用例，减少 API 调用与耗时。",
+    )
+    parser.add_argument(
+        "--limit", type=int, default=None, metavar="N",
+        help="每组最多评测 N 个用例（覆盖 --quick）。",
+    )
+    parser.add_argument(
+        "--group", choices=("holdout", "boundary", "both"), default="both",
+        help="选择评测的任务集：holdout(保留集) / boundary(边界集) / both(默认，两者都跑)。",
+    )
+    parser.add_argument(
+        "--rounds", type=int, default=3, metavar="N",
+        help="Coding Agent 自动改写提示词的最大重试轮数（默认 3）。",
+    )
+    parser.add_argument(
+        "--model", default=None, metavar="NAME",
+        help="覆盖 LLM 模型名（等价于设置环境变量 LLM_MODEL，如 gpt-5.6-luna）。",
+    )
+    parser.add_argument(
+        "--provider", choices=("openai", "moonshot", "ark", "openrouter"), default=None,
+        help="覆盖 LLM 提供商（等价于设置环境变量 LLM_PROVIDER，默认 openai）。",
+    )
+    parser.add_argument(
+        "--output", default=None, metavar="PATH",
+        help="把优化前后 + 人工对照的对比结果写入指定 JSON 文件（如 output/run.json）。",
+    )
+    parser.add_argument(
+        "--dry-run", action="store_true",
+        help="离线自检：只打印解析后的配置与选中用例数，不调用任何 LLM API。",
+    )
+    return parser
+
+
+if __name__ == "__main__":
+    args = _build_parser().parse_args()
+
+    # 命令行覆盖优先级高于环境变量：get_provider()/get_model() 均在调用时读取环境变量
+    if args.provider:
+        os.environ["LLM_PROVIDER"] = args.provider
+    if args.model:
+        os.environ["LLM_MODEL"] = args.model
+
+    limit = args.limit if args.limit is not None else (2 if args.quick else None)
+    groups = GROUPS if args.group == "both" else (args.group,)
+    cases = _select_cases(limit, groups=groups)
+
+    if args.dry_run:
+        # 离线路径：不触发任何网络请求，仅用于验证参数解析与用例选择
+        print("[dry-run] 解析后的运行配置（不调用 API）：")
+        print(f"  LLM 提供商 : {get_provider()}")
+        print(f"  LLM 模型   : {get_model()}")
+        print(f"  优化轮数   : {args.rounds}")
+        print(f"  任务集     : {args.group}")
+        print(f"  选中用例数 : {len(cases)}  -> {[c['id'] for c in cases]}")
+        print(f"  输出文件   : {args.output or '(不写文件)'}")
+        sys.exit(0)
+
+    try:
+        main(cases=cases, rounds=args.rounds, output=args.output)
+    except RuntimeError as e:
+        # 例如 API Key 未设置：给出清晰的人类可读错误，而非原始 traceback
+        print(f"\n[错误] {e}", file=sys.stderr)
+        sys.exit(1)

chapter9/prompt-auto-optimization/env.example

@@ -0,0 +1,24 @@
+# 复制本文件为 .env 并填入你的真实密钥（demo.py 会自动加载）
+
+# ===== 默认使用 OpenAI =====
+OPENAI_API_KEY=your-openai-api-key
+
+# 默认模型为 gpt-5.6-luna，可覆盖：
+# LLM_MODEL=gpt-5.6-luna
+
+# ===== 统一兜底：OpenRouter =====
+# 若上面 provider 的 Key 缺失，但设置了 OPENROUTER_API_KEY，则自动改走
+# OpenRouter，并把模型名映射到 openai/gpt-5.6-luna 等。
+# OPENROUTER_API_KEY=your-openrouter-api-key
+
+# ===== 可选：切换到国内兼容 OpenAI 接口的提供商 =====
+# LLM_PROVIDER 取值：openai(默认) / moonshot / ark / openrouter
+# LLM_PROVIDER=moonshot
+# MOONSHOT_API_KEY=your-moonshot-api-key
+
+# LLM_PROVIDER=ark
+# ARK_API_KEY=your-ark-key
+# ARK_MODEL=your-ark-endpoint-id     # 火山方舟需填推理接入点 ID
+
+# 采样温度（默认 0，保证结果可复现）
+# LLM_TEMPERATURE=0

chapter9/prompt-auto-optimization/evaluate.py

@@ -0,0 +1,126 @@
+"""
+评测器：给定一份 system prompt，在用例集上运行 Agent 并判定每个用例是否被"正确处理"。
+
+判定规则（结合确定性规则 + LLM-as-judge）：
+- 用例标注了 should_transfer（正确行为是否应转接人工）。
+- 若 should_transfer=True：  正确 ⇔ Agent 确实转接了。
+- 若 should_transfer=False： 正确 ⇔ Agent 没有转接，且 LLM 裁判确认它按 rubric 妥善处理了请求
+                             （即真的解释了政策 / 办理了业务，而不是敷衍或答非所问）。
+  之所以对"不转接"再加一道 LLM 裁判，是为了防止 Agent 只是"没转接但也没好好处理"蒙混过关。
+"""
+
+import json
+from config import get_client, get_model, get_temperature, record_completion
+from airline_env import run_agent
+
+
+def _judge_handled(user_message: str, rubric: str, agent_reply: str) -> dict:
+    """LLM-as-judge：Agent 没转接的情况下，是否按 rubric 妥善处理了请求。"""
+    client = get_client()
+    model = get_model()
+    prompt = f"""你是严格的客服质检员。请判断客服 Agent 的回复是否妥善处理了乘客请求。
+
+【乘客请求】
+{user_message}
+
+【合格标准(rubric)】
+{rubric}
+
+【Agent 的回复】
+{agent_reply}
+
+请只输出一个 JSON：{{"handled": true 或 false, "reason": "简短理由"}}
+其中 handled=true 表示 Agent 的回复实质满足了合格标准。"""
+    resp = record_completion(client, kind="llm_judge",
+        model=model,
+        messages=[{"role": "user", "content": prompt}],
+        temperature=get_temperature(),
+        response_format={"type": "json_object"},
+    )
+    try:
+        verdict = json.loads(resp.choices[0].message.content)
+        return {
+            "handled": bool(verdict.get("handled", False)),
+            "reason": str(verdict.get("reason", "")),
+        }
+    except (json.JSONDecodeError, TypeError):
+        return {"handled": False, "reason": "judge returned invalid JSON"}
+
+
+def evaluate_case(system_prompt: str, case: dict, verbose: bool = False) -> dict:
+    """评测单个用例，返回结果 dict。"""
+    result = run_agent(system_prompt, case["user"])
+    transferred = result["transferred"]
+    should_transfer = case["should_transfer"]
+
+    handled = None
+    if should_transfer:
+        correct = transferred
+        note = "应转接：" + ("已转接 ✓" if transferred else "未转接 ✗")
+    else:
+        if transferred:
+            correct = False
+            note = "不应转接：却转接了 ✗（过度转接）"
+        else:
+            judge = _judge_handled(case["user"], case["rubric"], result["final_text"])
+            handled = judge["handled"]
+            judge_reason = judge["reason"]
+            correct = handled
+            note = "不应转接：未转接且妥善处理 ✓" if handled else "不应转接：未转接但处理不当 ✗"
+
+    out = {
+        "id": case["id"],
+        "group": case["group"],
+        "correct": correct,
+        "transferred": transferred,
+        "should_transfer": should_transfer,
+        "note": note,
+        "final_text": result["final_text"],
+        "transfer_reason": result["transfer_reason"],
+        "tool_calls": result["tool_calls"],
+        "handled": handled,
+        "judge_reason": locals().get("judge_reason"),
+        "rubric": case["rubric"],
+        "user": case["user"],
+    }
+    if verbose:
+        icon = "✓" if correct else "✗"
+        print(f"  [{icon}] {case['id']:<16} {note}")
+        if transferred:
+            print(f"        转接原因: {result['transfer_reason']}")
+        else:
+            preview = (result["final_text"] or "").replace("\n", " ")[:80]
+            print(f"        回复: {preview}...")
+    return out
+
+
+def evaluate_prompt(system_prompt: str, label: str = "", verbose: bool = True, cases=None) -> dict:
+    """在全部用例上评测一份 prompt，返回分组正确率与明细。
+
+    cases 为 None 时评测全部用例；也可传入用例子集（如 --quick 模式）以控制成本。
+    """
+    from airline_env import CASES
+
+    if cases is None:
+        cases = CASES
+
+    if verbose and label:
+        print(f"\n>>> 评测 [{label}]")
+    results = []
+    for case in cases:
+        results.append(evaluate_case(system_prompt, case, verbose=verbose))
+
+    def _acc(group):
+        rows = [r for r in results if r["group"] == group]
+        n = len(rows)
+        c = sum(1 for r in rows if r["correct"])
+        return c, n
+
+    holdout_c, holdout_n = _acc("holdout")
+    boundary_c, boundary_n = _acc("boundary")
+    return {
+        "label": label,
+        "holdout": (holdout_c, holdout_n),
+        "boundary": (boundary_c, boundary_n),
+        "results": results,
+    }

chapter9/prompt-auto-optimization/learning_signal.py

@@ -0,0 +1,111 @@
+"""Turn evaluated customer-service trajectories into a prompt-learning signal."""
+
+from __future__ import annotations
+
+from collections import defaultdict
+from typing import Any, Dict
+
+
+DIMENSIONS = ("rule_compliance", "task_resolution", "compliant_flexibility")
+
+
+def case_dimensions(result: Dict[str, Any]) -> Dict[str, Dict[str, Any]]:
+    """Derive the three dimensions used by Experiment 9-3.
+
+    The full project would consume Experiment 9-1 reports.  This compact
+    airline simulation exposes the same dimensions from its deterministic
+    labels and the handled/not-handled judge result.
+    """
+    should_transfer = bool(result["should_transfer"])
+    transferred = bool(result["transferred"])
+    boundary = result["group"] == "boundary"
+    correct = bool(result["correct"])
+
+    if should_transfer and not transferred:
+        compliance = {
+            "verdict": "fail",
+            "evidence": "Required escalation boundary was ignored",
+        }
+    else:
+        compliance = {
+            "verdict": "pass",
+            "evidence": "No forbidden action or missed mandatory escalation was observed",
+        }
+
+    resolution = {
+        "verdict": "pass" if correct else "fail",
+        "evidence": result.get("note", ""),
+    }
+
+    if boundary:
+        flexible = not transferred and bool(result.get("handled"))
+        flexibility = {
+            "verdict": "pass" if flexible else "fail",
+            "evidence": (
+                "Explained policy and supplied an allowed alternative"
+                if flexible
+                else "Policy dispute was transferred or left without a useful compliant alternative"
+            ),
+        }
+    else:
+        flexibility = {
+            "verdict": "pass",
+            "evidence": "This case does not require a blocked-path alternative",
+        }
+    return {
+        "rule_compliance": compliance,
+        "task_resolution": resolution,
+        "compliant_flexibility": flexibility,
+    }
+
+
+def diagnose_failures(evaluation: Dict[str, Any]) -> Dict[str, Any]:
+    """Aggregate failed cases into an evidence-bearing change request."""
+    failed_by_dimension: Dict[str, list[Dict[str, str]]] = defaultdict(list)
+    all_case_reports = []
+    for result in evaluation.get("results", []):
+        dimensions = case_dimensions(result)
+        all_case_reports.append({"case_id": result["id"], "dimensions": dimensions})
+        for dimension, verdict in dimensions.items():
+            if verdict["verdict"] == "fail":
+                failed_by_dimension[dimension].append({
+                    "case_id": result["id"],
+                    "evidence": verdict["evidence"],
+                })
+
+    source_ids = sorted({
+        item["case_id"]
+        for failures in failed_by_dimension.values()
+        for item in failures
+    })
+    boundary_ids = [
+        item["case_id"]
+        for item in failed_by_dimension.get("compliant_flexibility", [])
+    ]
+    diagnosis = (
+        "The prompt over-escalates policy disputes. Preserve mandatory escalation for explicit "
+        "human requests and safety emergencies, but require policy explanation and an allowed "
+        "alternative before transfer in ordinary disputes."
+        if boundary_ids
+        else "No repeated prompt-level boundary failure was detected."
+    )
+    return {
+        "source_case_ids": source_ids,
+        "scope": "system_prompt.transfer_policy",
+        "dimensions": {dimension: failed_by_dimension.get(dimension, []) for dimension in DIMENSIONS},
+        "diagnosis": diagnosis,
+        "case_reports": all_case_reports,
+    }
+
+
+def format_learning_signal(report: Dict[str, Any]) -> str:
+    lines = [
+        f"Scope: {report['scope']}",
+        f"Source cases: {', '.join(report['source_case_ids']) or 'none'}",
+        f"Diagnosis: {report['diagnosis']}",
+    ]
+    for dimension in DIMENSIONS:
+        failures = report["dimensions"].get(dimension, [])
+        lines.append(f"{dimension}: {len(failures)} failure(s)")
+        lines.extend(f"- {item['case_id']}: {item['evidence']}" for item in failures)
+    return "\n".join(lines)

chapter9/prompt-auto-optimization/prompts/system_prompt.txt

@@ -0,0 +1,20 @@
+你是「云舒航空」的在线客服 Agent，负责通过对话帮助乘客处理订票、改签、退票、行李、选座等售后事务。
+
+# 你的能力范围
+你可以调用以下工具为乘客服务：
+- lookup_reservation：根据订单号查询订单详情
+- change_flight：为乘客办理改签
+- get_refund_policy：查询退票政策
+- get_baggage_policy：查询行李额与逾重费政策
+- change_seat：为乘客办理选座 / 换座
+- transfer_to_human：转接人工客服
+
+# 服务原则
+1. 礼貌、专业、简洁地回应乘客，全程使用中文。
+2. 优先使用工具查询真实信息后再回答，不要编造政策或订单信息。
+3. 关于人工转接：仅当请求无法在你的行动范围内处理时，才转接人工客服。
+4. 客户满意度是第一位的。为避免激化矛盾，当乘客表达不满、投诉，或提出你无法完全满足的诉求（例如要求超出政策的退款、赔偿、免费或费用减免）时，应当及时转接人工客服来处理，不要自行与乘客争辩政策。
+
+# 回复要求
+- 如需转接，请调用 transfer_to_human 工具并说明原因。
+- 如能自行处理，请在完成必要的工具调用后，用一段简洁的话回复乘客。

chapter9/prompt-auto-optimization/prompts/system_prompt_manual.txt

@@ -0,0 +1,24 @@
+你是「云舒航空」的在线客服 Agent，负责通过对话帮助乘客处理订票、改签、退票、行李、选座等售后事务。
+
+# 你的能力范围
+你可以调用以下工具为乘客服务：
+- lookup_reservation：根据订单号查询订单详情
+- change_flight：为乘客办理改签
+- get_refund_policy：查询退票政策
+- get_baggage_policy：查询行李额与逾重费政策
+- change_seat：为乘客办理选座 / 换座
+- transfer_to_human：转接人工客服
+
+# 服务原则
+1. 礼貌、专业、简洁地回应乘客，全程使用中文。
+2. 优先使用工具查询真实信息后再回答，不要编造政策或订单信息。
+3. 关于人工转接：只有在下面两种情况下才可以转接人工客服——
+   (a) 乘客明确、直接地要求与人工客服对话；
+   (b) 出现紧急安全或人身健康风险，需要人工立即介入。
+   除此之外的所有咨询、办理、纠纷，都应由你自己处理到底。
+4. 遇到政策争议（例如乘客要求超出政策的退款、免费、豁免费用）时，绝不能因为对方不满就一转了之。
+   你应当先用工具查清相关政策，再耐心、共情地向乘客解释规则，并主动提供政策允许范围内的可行替代方案。
+
+# 回复要求
+- 只有在符合第 3 条的两种情况时，才调用 transfer_to_human 工具，并说明原因。
+- 其余情况请在完成必要的工具调用后，用一段简洁、专业、有同理心的话回复乘客。

chapter9/prompt-auto-optimization/release_gate.py

@@ -0,0 +1,54 @@
+"""Candidate manifest and release gate for prompt updates."""
+
+from __future__ import annotations
+
+from typing import Any, Dict
+
+
+def build_candidate_manifest(
+    optimization: Dict[str, Any], learning_signal: Dict[str, Any]
+) -> Dict[str, Any]:
+    return {
+        "artifact_type": "system_prompt_patch",
+        "source_case_ids": list(learning_signal.get("source_case_ids", [])),
+        "scope": learning_signal.get("scope", "system_prompt"),
+        "rationale": optimization.get("rationale") or learning_signal.get("diagnosis", ""),
+        "diff": optimization.get("diff", ""),
+        "edits": list(optimization.get("edits", [])),
+        "target_rule": "transfer only on explicit human request or urgent safety event; otherwise explain policy and seek compliant alternatives",
+        "status": "candidate",
+    }
+
+
+def evaluate_release_gate(
+    before: Dict[str, Any], after: Dict[str, Any], manifest: Dict[str, Any]
+) -> Dict[str, Any]:
+    holdout_before, holdout_total = before["holdout"]
+    holdout_after, _ = after["holdout"]
+    boundary_before, boundary_total = before["boundary"]
+    boundary_after, _ = after["boundary"]
+
+    checks = {
+        "patch_is_nonempty": bool(manifest.get("diff", "").strip()),
+        "patch_is_auditable_old_to_new_edit": bool(manifest.get("edits")) and all(
+            isinstance(edit, dict)
+            and isinstance(edit.get("old_str"), str) and bool(edit["old_str"])
+            and isinstance(edit.get("new_str"), str) and bool(edit["new_str"])
+            for edit in manifest.get("edits", [])
+        ),
+        "source_cases_are_recorded": bool(manifest.get("source_case_ids")),
+        "holdout_did_not_regress": holdout_after >= holdout_before,
+        "boundary_improved": boundary_after > boundary_before,
+    }
+    accepted = all(checks.values())
+    return {
+        "decision": "release_to_canary" if accepted else "reject_candidate",
+        "accepted": accepted,
+        "checks": checks,
+        "metrics": {
+            "holdout_before": [holdout_before, holdout_total],
+            "holdout_after": [holdout_after, holdout_total],
+            "boundary_before": [boundary_before, boundary_total],
+            "boundary_after": [boundary_after, boundary_total],
+        },
+    }

chapter9/prompt-auto-optimization/requirements.txt

@@ -0,0 +1,2 @@
+openai>=1.30.0
+python-dotenv>=1.0.0

chapter9/prompt-auto-optimization/run_experiment_9_3.py

@@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+"""Run the full Experiment 9-3 campaign and save canonical evidence."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+from datetime import datetime, timezone
+from pathlib import Path
+
+from airline_env import CASES
+from demo import main as run_campaign
+
+
+ROOT = Path(__file__).resolve().parent
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--provider", choices=("openrouter", "moonshot", "ark", "openai"), default="openrouter")
+    parser.add_argument("--model", default="openai/gpt-4o-mini")
+    parser.add_argument("--rounds", type=int, default=3)
+    parser.add_argument("--output-dir", type=Path)
+    args = parser.parse_args()
+    os.environ["LLM_PROVIDER"] = args.provider
+    os.environ["LLM_MODEL"] = args.model
+    stamp = datetime.now(timezone.utc).strftime("real_%Y%m%dT%H%M%SZ")
+    output_dir = args.output_dir or ROOT / "validation" / stamp
+    output_dir.mkdir(parents=True, exist_ok=False)
+    evidence_path = output_dir / "evidence.json"
+    summary = run_campaign(cases=CASES, rounds=args.rounds, output=str(evidence_path))
+    payload = evidence_path.read_text(encoding="utf-8")
+    (ROOT / "validation").mkdir(exist_ok=True)
+    (ROOT / "validation" / "latest.json").write_text(payload, encoding="utf-8")
+    print(json.dumps({
+        "evidence": str(evidence_path.relative_to(ROOT)),
+        "sha256": hashlib.sha256(payload.encode()).hexdigest(),
+        "execution_accepted": summary["acceptance"]["execution_accepted"],
+        "all_manuscript_result_claims_observed": summary["acceptance"]["all_manuscript_result_claims_observed"],
+        "release_decision": summary["release_gate"]["decision"],
+        "usage": summary["usage"],
+    }, ensure_ascii=False, indent=2))
+    return 0 if summary["acceptance"]["execution_accepted"] else 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter9/prompt-auto-optimization/test_non_dict_edit.py

@@ -0,0 +1,68 @@
+"""Non-dict items in edits must not cause AttributeError or roll back valid edits in optimize_prompt."""
+import tempfile
+from unittest.mock import MagicMock, patch
+from coding_agent import _apply_edits_from_args, optimize_prompt
+
+
+def test_string_edit_item_skipped_with_warning():
+    working, applied, errors, warnings, edits = _apply_edits_from_args(
+        "hello world",
+        {"edits": ["bad", {"old_str": "hello", "new_str": "hi"}]},
+    )
+    assert working == "hi world"
+    assert applied == 1
+    assert errors == []
+    assert any("跳过非对象" in w for w in warnings)
+    assert len(edits) == 2
+
+
+def test_null_edit_item_skipped_with_warning():
+    working, applied, errors, warnings, _ = _apply_edits_from_args(
+        "hello world",
+        {"edits": [None]},
+    )
+    assert working == "hello world"
+    assert applied == 0
+    assert errors == []
+    assert any("跳过非对象" in w for w in warnings)
+
+
+def test_null_edits_list_still_empty():
+    working, applied, errors, warnings, edits = _apply_edits_from_args(
+        "hello world", {"edits": None}
+    )
+    assert working == "hello world"
+    assert applied == 0
+    assert errors == []
+    assert warnings == []
+    assert edits == []
+
+
+def test_optimize_prompt_applies_valid_edits_when_non_dict_items_present():
+    """optimize_prompt must write valid edits to disk even if non-dict items are in the edits array."""
+    with tempfile.NamedTemporaryFile("w+", delete=False, encoding="utf-8") as f:
+        f.write("hello world")
+        prompt_file = f.name
+
+    mock_tool_call = MagicMock()
+    mock_tool_call.id = "tc_1"
+    mock_tool_call.function.arguments = '{"edits": ["invalid_string_item", {"old_str": "hello", "new_str": "greetings"}]}'
+
+    mock_msg = MagicMock()
+    mock_msg.tool_calls = [mock_tool_call]
+    mock_msg.content = None
+
+    mock_response = MagicMock()
+    mock_response.choices = [MagicMock(message=mock_msg)]
+
+    mock_client = MagicMock()
+    mock_client.chat.completions.create.return_value = mock_response
+
+    with patch("coding_agent.get_client", return_value=mock_client), \
+         patch("coding_agent.get_model", return_value="gpt-4o"):
+        res = optimize_prompt(prompt_file, feedback="test feedback", verbose=False)
+
+    assert res["after"] == "greetings world"
+    with open(prompt_file, "r", encoding="utf-8") as f:
+        content = f.read()
+    assert content == "greetings world"

chapter9/prompt-auto-optimization/test_release_gate_non_dict_edit.py

@@ -0,0 +1,16 @@
+import pytest
+from release_gate import evaluate_release_gate
+
+
+def test_evaluate_release_gate_non_dict_edit_item():
+    before = {"holdout": (5, 10), "boundary": (3, 5)}
+    after = {"holdout": (5, 10), "boundary": (4, 5)}
+    manifest = {
+        "diff": "diff text",
+        "edits": [None, "string_edit", {"old_str": "a", "new_str": "b"}],
+        "source_case_ids": ["1"],
+    }
+    result = evaluate_release_gate(before, after, manifest)
+    assert result["accepted"] is False
+    assert result["checks"]["patch_is_auditable_old_to_new_edit"] is False
+    assert result["decision"] == "reject_candidate"

chapter9/prompt-auto-optimization/tests/conftest.py

@@ -0,0 +1,9 @@
+"""Test import bootstrap for the prompt-auto-optimization experiment."""
+
+from pathlib import Path
+import sys
+
+
+EXPERIMENT_ROOT = Path(__file__).resolve().parents[1]
+if str(EXPERIMENT_ROOT) not in sys.path:
+    sys.path.insert(0, str(EXPERIMENT_ROOT))

chapter9/prompt-auto-optimization/tests/test_apply_one_null.py

@@ -0,0 +1,21 @@
+from coding_agent import _apply_one
+
+
+def test_apply_one_null_old_str():
+    content, err = _apply_one("hello world", None, "x")
+    assert content == "hello world"
+    assert err is not None
+    assert "null" in err
+
+
+def test_apply_one_null_new_str():
+    content, err = _apply_one("hello world", "hello", None)
+    assert content == "hello world"
+    assert err is not None
+    assert "null" in err
+
+
+def test_apply_one_normal():
+    content, err = _apply_one("hello world", "hello", "hi")
+    assert err is None
+    assert content == "hi world"

chapter9/prompt-auto-optimization/tests/test_baggage_policy_null.py

@@ -0,0 +1,22 @@
+import json
+
+from airline_env import _run_tool
+
+
+def test_baggage_policy_null_cabin():
+    # 模型显式传 {"cabin": null}：应回退到经济舱默认，而不是 TypeError
+    args = json.loads('{"cabin": null}')
+    result = json.loads(_run_tool("get_baggage_policy", args))
+    assert result["cabin"] == "经济舱"
+    assert result["free_allowance"] == "20kg"
+
+
+def test_baggage_policy_missing_cabin():
+    result = json.loads(_run_tool("get_baggage_policy", {}))
+    assert result["cabin"] == "经济舱"
+    assert result["free_allowance"] == "20kg"
+
+
+def test_baggage_policy_business_cabin():
+    result = json.loads(_run_tool("get_baggage_policy", {"cabin": "商务舱"}))
+    assert result["free_allowance"] == "30kg"

chapter9/prompt-auto-optimization/tests/test_edits_list_null.py

@@ -0,0 +1,27 @@
+from coding_agent import _apply_edits_from_args, _apply_one
+
+
+def test_null_edits_like_empty():
+    working, applied, errors, warnings, edits = _apply_edits_from_args("hello world", {"edits": None})
+    assert working == "hello world"
+    assert applied == 0
+    assert errors == []
+    assert warnings == []
+    assert edits == []
+
+
+def test_apply_edits_normal():
+    working, applied, errors, warnings, edits = _apply_edits_from_args(
+        "hello world",
+        {"edits": [{"old_str": "hello", "new_str": "hi"}]},
+    )
+    assert working == "hi world"
+    assert applied == 1
+    assert errors == []
+    assert warnings == []
+    assert len(edits) == 1
+
+
+def test_apply_one_still_rejects_null_strings():
+    content, err = _apply_one("hello", None, "x")
+    assert err is not None

chapter9/prompt-auto-optimization/tests/test_learning_and_release.py

@@ -0,0 +1,57 @@
+import unittest
+
+from learning_signal import diagnose_failures, format_learning_signal
+from release_gate import build_candidate_manifest, evaluate_release_gate
+
+
+def evaluation(holdout=(2, 2), boundary=(0, 2)):
+    return {
+        "holdout": holdout,
+        "boundary": boundary,
+        "results": [
+            {
+                "id": "B1",
+                "group": "boundary",
+                "correct": False,
+                "transferred": True,
+                "should_transfer": False,
+                "handled": None,
+                "note": "不应转接：却转接了",
+            }
+        ],
+    }
+
+
+class LearningAndReleaseTest(unittest.TestCase):
+    def test_diagnosis_comes_from_failed_case(self):
+        report = diagnose_failures(evaluation())
+        self.assertEqual(["B1"], report["source_case_ids"])
+        self.assertEqual("system_prompt.transfer_policy", report["scope"])
+        self.assertEqual("B1", report["dimensions"]["compliant_flexibility"][0]["case_id"])
+        self.assertIn("Source cases: B1", format_learning_signal(report))
+
+    def test_release_requires_improvement_and_no_regression(self):
+        signal = diagnose_failures(evaluation())
+        manifest = build_candidate_manifest({
+            "diff": "+ new rule", "rationale": "narrow transfer",
+            "edits": [{"old_str": "old rule", "new_str": "new rule"}],
+        }, signal)
+        accepted = evaluate_release_gate(evaluation(), evaluation(boundary=(1, 2)), manifest)
+        self.assertTrue(accepted["accepted"])
+        self.assertEqual("release_to_canary", accepted["decision"])
+
+        regressed = evaluate_release_gate(
+            evaluation(holdout=(2, 2)), evaluation(holdout=(1, 2), boundary=(1, 2)), manifest
+        )
+        self.assertFalse(regressed["accepted"])
+        self.assertFalse(regressed["checks"]["holdout_did_not_regress"])
+
+    def test_empty_patch_is_rejected(self):
+        signal = diagnose_failures(evaluation())
+        manifest = build_candidate_manifest({"diff": "", "rationale": "none", "edits": []}, signal)
+        decision = evaluate_release_gate(evaluation(), evaluation(boundary=(1, 2)), manifest)
+        self.assertEqual("reject_candidate", decision["decision"])
+
+
+if __name__ == "__main__":
+    unittest.main()