liqiang/ai-agent-book
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
Details
dependency resolution / resolve (3.11) (push) Canceled after 0s
Details
dependency resolution / resolve (3.13) (push) Canceled after 0s
Details
deploy-pages / build (push) Canceled after 0s
Details
deploy-pages / deploy (push) Canceled after 0s
Details
i18n consistency check / check (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
Details
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
Details
web-search-agent tests / test (push) Canceled after 0s
Details
web-search-agent tests / agentbook (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
liqiang
2026-08-20 13:12:50 +00:00
commit b119135836
10275 changed files with 3284984 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
chapter5/log-diagnosis/.gitignore
+11
View File
@@ -0,0 +1,11 @@
# 生成产物
output/
# Python
__pycache__/
*.pyc
.venv/
venv/
# 环境变量
.env
chapter5/log-diagnosis/README.md
+369
View File
@@ -0,0 +1,369 @@
# Experiment 5-8: Production Log Diagnosis / 实验 5-8：生产日志的智能诊断系统
> Companion lab for *AI Agents in Depth*, Chapter 5 — diagnose trajectories + architecture + PRD → structured report → regression tests → real replay → GitHub Issues through the official MCP server.
> 《深入理解 AI Agent》第 5 章：诊断 Agent 读轨迹/架构/PRD，定位根因、生成回归测试、真实重放验证，并通过官方 GitHub MCP 创建 Issue。
← [Chapter 5 index / 返回第 5 章目录](../README.md)
---
## Canonical manuscript experiment / 正文正式实验
The manuscript contract is satisfied by
[`validation/runs/exp5-8-live-http-mcp-20260730-053403/manifest.json`](validation/runs/exp5-8-live-http-mcp-20260730-053403/manifest.json),
not by the smaller deterministic demo described later in this README. The
canonical campaign:
- recorded two trajectories from a real local HTTP subprocess, including raw
HTTP results and measured latency;
- made two live `doubao-seed-1-6-250615` calls to diagnose the failures and
generate executable tests tied to trajectory IDs and turn numbers;
- executed all three generated tests against both HTTP implementations: every
test failed on the buggy service and passed after the fix; and
- called `issue_write(method=create)` on the official
`github/github-mcp-server`, creating
[Issue #502](https://github.com/bojieli/ai-agent-book/issues/502).
All nine acceptance gates are true and the manifest SHA-256 is
`68e09e7c8b4fc100e0612a6f81978079c393a822025bfa794b6dda85134a5813`.
Raw provider calls, live replays, the generated tests, and the credential-free
MCP receipt are retained beside the manifest.
正文合同由
[`validation/runs/exp5-8-live-http-mcp-20260730-053403/manifest.json`](validation/runs/exp5-8-live-http-mcp-20260730-053403/manifest.json)
中的正式活动满足，而不是下文较小的确定性演示。该活动从真实本地 HTTP
子进程采集带原始响应和实测延迟的轨迹；用真实
`doubao-seed-1-6-250615` 调用完成诊断和可执行测试生成；让三个测试在有
缺陷实现上全部失败、修复实现上全部通过；最后通过官方
`github/github-mcp-server` 的 `issue_write(method=create)` 创建了
[Issue #502](https://github.com/bojieli/ai-agent-book/issues/502)。九项验收门禁全部为
`true`，原始模型回执、重放、测试与去凭据 MCP 回执均随 manifest 保留。
The sections below document the compact teaching/CI implementation. Its
default GitHub sink is intentionally a mock; `--create-issue` selects its live
MCP branch. Running that legacy path alone is not evidence for the canonical
experiment.
下文记录的是适合教学和 CI 的小型实现；其 GitHub 默认输出有意采用 mock，
`--create-issue` 才进入真实 MCP 分支。只运行该旧路径不能作为正式实验完成证据。
---
## English
### Purpose
Production Agents emit large **trajectory logs**. Finding issues, root causes, and building regression tests is expensive. This lab automates:
**Read trajectory set + architecture + PRD → locate issues, structured report → generate regression cases → replay framework executes for real → (mock) GitHub Issues via MCP.**
### Diagnosis pipeline
```
data/trajectories.jsonl (production trajectories with known issues)
data/architecture.md (system architecture) ┐
data/PRD.md (product requirements) ├─► [LLM] diagnose() structured issue report
┘ │
▼
[LLM] gen_test_cases() regression cases (trajectory IDs + turns)
│
▼
replay.py replay framework
(A) unfixed SUT → FAIL (reproduce bug)
(B) fixed SUT → PASS (verify fix)
│
▼
github_mcp.py (mock) print/write GitHub Issues
```
- `diagnoser.py`: diagnosis Agent; two real OpenAI calls (default gpt-5.6-luna, JSON mode).
- `sut.py`: **deterministic** system-under-test simulator. `fixed=False` reproduces bugs; `fixed=True` fixed behavior.
- `replay.py`: regression replay. Trajectory input → replay `sut` → evaluate asserts on new trajectory (4 built-in assert DSL kinds).
- `github_mcp.py`: GitHub Issue create; default mock (print + `output/github_issues.json`).
### Seeded known issues (Agent should find)
| Trajectory | Issue | PRD violated | Module |
|------|------|----------|----------|
| T-1001 / T-1002 | Skipped mandatory `verify_refund_eligibility` before refund | R1 (P0) | order_service |
| T-1002 | `process_refund` **repeated failures**, no backoff, false success | R2 (P0) | payment_service |
| T-1003 | `check_stock` latency 8300ms **timeout without degrade** | R3 (P1) | inventory_service |
| T-1004 | Healthy trajectory (control) | — | — |
### Run
```bash
# From the repository root: use the shared Chapter 5 environment
uv sync --locked --python 3.12 --extra ch5
# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat
# pip fallback when uv is not installed:
# python -m pip install -e ".[ch5]"
cd chapter5/log-diagnosis
# Single-project compatibility path, still supported during migration:
# python -m pip install -r requirements.txt
cp env.example .env # OPENAI_API_KEY (default gpt-5.6-luna); or OPENROUTER_API_KEY
python demo.py # full pipeline (two real LLM calls)
```
`demo.py` once: read trajectories → diagnosis report → regression cases → replay pass/fail → (mock) GitHub Issue.
Common flags (`python demo.py -h`):
- `--smoke`: **no-API smoke**—skip LLM; built-in diagnosis + replay + GitHub mock only (exit 0 if green). CI / no key.
- `--model gpt-5.6`: override model (same as `OPENAI_MODEL`).
- `--data-dir DIR`: input dir with `trajectories.jsonl` + `architecture.md` + `PRD.md` (default `data/`).
- `--output FILE`: mock Issue write path (default `output/github_issues.json`).
- `--create-issue`: **real** Issues via MCP (`GITHUB_TOKEN` + `GITHUB_REPO`; falls back to mock if missing).
- `--no-github`: skip step 4.
### Sample real output (excerpt)
Diagnosis found all 3 seeded issue classes:
```
[问题 1] 未进行退款资格校验
优先级 : P0 模块: order_service PRD: R1
轨迹 : ['T-1001', 'T-1002'] 关键轮次: [3]
[问题 2] 支付重试机制未正确实现
优先级 : P0 模块: payment_service PRD: R2
[问题 3] 库存查询延迟未降级处理
优先级 : P1 模块: inventory_service PRD: R3
```
Replay **actually runs** cases (reproduce then verify fix):
```
(A) 对『线上未修复』系统重放 —— 期望复现 bug（FAIL）
[FAIL] RT-001 (T-1001) 工具 verify_refund_eligibility 缺失
[FAIL] RT-002 (T-1002) process_refund 调用 3 次, 失败 3 次, 末次失败
[FAIL] RT-003 (T-1003) check_stock 最大延迟 8300ms, 阈值 5000ms
(B) 对『修复后』系统重放 —— 期望修复被验证（PASS）
[PASS] RT-001 (T-1001) 工具 verify_refund_eligibility 出现
[PASS] RT-002 (T-1002) process_refund 调用 2 次, 失败 1 次, 末次成功
[PASS] RT-003 (T-1003) check_stock 最大延迟 400ms, 阈值 5000ms
小结：复现 bug 3/3 条；修复后通过 3/3 条。
```
Mock Issue example written to `output/github_issues.json`:
```
title : [P0][order_service] 未进行退款资格校验
labels : ['module:order_service', 'priority:critical', 'auto-diagnosis']
body : ## 问题描述 ... ## 关联回归测试用例 - RT-001 (轨迹 T-1001 第 3 轮) ...
```
### Regression assert DSL (built into replay)
Generated cases must use one of:
- `step_present` `{tool}`: tool must appear (e.g. mandatory pre-check).
- `tool_succeeds` `{tool}`: tool eventually succeeds; no “many fails then fake success”.
- `latency_under` `{tool, threshold_ms}`: single-call latency under threshold.
- `final_status_is` `{value}`: final task status equals value.
### Adapt / extend
- **Model**: `OPENAI_MODEL` or `python demo.py --model <name>`. Default `gpt-5.6-luna`, JSON mode.
- **Provider**: official `openai` SDK + `OPENAI_BASE_URL` + provider key/model, e.g.:
```bash
export OPENAI_BASE_URL=https://api.moonshot.cn/v1
export OPENAI_API_KEY=your-openai-api-key
export OPENAI_MODEL=kimi-k3
python demo.py
```
- **Logs**: replace `data/trajectories.jsonl` (`trajectory_id / task / task_input / turns[] / final_status`; turns with `module/tool/input/output/status/latency_ms`); update `architecture.md` / `PRD.md`. Adjust `sut.py` / `replay.py` if fields differ.
- **Real GitHub MCP**: next section; `GITHUB_TOKEN` + `GITHUB_REPO` + `--create-issue`.
### Real GitHub MCP (token required; default mock)
Default is mock; `--create-issue` goes live. Implementation in `github_mcp._create_issues_via_mcp()`: MCP client (`mcp` SDK, stdio) to official GitHub MCP Server, `create_issue` with `build_issue()` fields.
1. GitHub PAT with `repo` → `.env` `GITHUB_TOKEN`; set `GITHUB_REPO=owner/repo`.
2. Machine can start official GitHub MCP Server (default Docker `ghcr.io/github/github-mcp-server`); override with `GITHUB_MCP_COMMAND` (token as `GITHUB_PERSONAL_ACCESS_TOKEN`).
3. The root `ch5` extra includes the MCP SDK; the compatibility path above also keeps the old project-local install available.
4. `python demo.py --create-issue`. Missing token/repo → tip + mock fallback.
### Limitations
- `sut.py` is a **deterministic sim** so replay can truly pass/fail; real systems need real stubs/replay.
- Diagnosis quality depends on LLM. gpt-5.6-luna stably finds R1/R2/R3 on this data; it often splits R1 across T-1001 and T-1002 (4 issues vs 3 merged). For payment retry asserts it may choose `final_status_is:failed` instead of `tool_succeeds` (fixed SUT ends success → FAIL after fix, e.g. 3/4). Tool names sometimes get module prefixes incompatible with bare-name matching (further fails). `--smoke` built-in cases are deterministic 3/3.
- GitHub create is mock unless `--create-issue` with token + repo + MCP server.
- Trajectory schema is simplified vs production (tokens, sub-agent trees, etc.).
---
## 中文
### 目的
生产环境的 Agent 会产生大量**轨迹日志**（trajectory）。从中识别问题、定位根因、构建回归测试成本很高。
本实验让一个诊断 Agent 自动完成这条流水线：
**读轨迹集合 + 架构文档 + PRD → 定位问题、生成结构化报告 → 生成回归测试用例 → 重放框架真正执行验证 → (mock) 通过 MCP 对接 GitHub 创建 Issue。**
### 诊断流水线
```
data/trajectories.jsonl (含已知问题的生产轨迹)
data/architecture.md (系统架构) ┐
data/PRD.md (产品需求) ├─► [LLM] diagnose() 结构化问题报告(优先级/模块/描述/建议)
┘ │
▼
[LLM] gen_test_cases() 回归测试用例(引用轨迹ID+交互轮次)
│
▼
replay.py 重放框架 ── 对同一输入重放被测系统并断言
(A) 未修复系统 → FAIL(复现bug)
(B) 修复后系统 → PASS(验证修复)
│
▼
github_mcp.py (mock) 渲染并打印/落盘 GitHub Issue
```
- `diagnoser.py`：诊断 Agent，两次真实调用 OpenAI（默认 gpt-5.6-luna，JSON 模式）。
- `sut.py`：被测系统的**确定性仿真器**。`fixed=False` 复现线上 bug，`fixed=True` 模拟修复后行为。
- `replay.py`：回归测试重放框架。取轨迹输入 → 重放 `sut` → 在新轨迹上求值断言（内置 4 种断言 DSL）。
- `github_mcp.py`：GitHub Issue 创建，默认 mock（打印 + 写 `output/github_issues.json`）。
### 预置的已知问题（Agent 应能定位）
| 轨迹 | 问题 | 违反 PRD | 定位模块 |
|------|------|----------|----------|
| T-1001 / T-1002 | 退款前**跳过**了强制的 `verify_refund_eligibility` 校验 | R1 (P0) | order_service |
| T-1002 | `process_refund` **反复失败**、无退避、且最终误报成功 | R2 (P0) | payment_service |
| T-1003 | `check_stock` 延迟 8300ms **超时未降级** | R3 (P1) | inventory_service |
| T-1004 | 正常轨迹（对照组，无问题） | — | — |
### 运行
```bash
# 在仓库根目录使用统一的第 5 章环境
uv sync --locked --python 3.12 --extra ch5
# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat
# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch5]"
cd chapter5/log-diagnosis
# 迁移期间仍支持单项目兼容路径：
# python -m pip install -r requirements.txt
cp env.example .env # 填入 OPENAI_API_KEY（模型默认 gpt-5.6-luna）；未配置时设 OPENROUTER_API_KEY 自动改走 OpenRouter
python demo.py # 完整流程（两次真实 LLM 调用）
```
`demo.py` 一次跑完：读轨迹 → 诊断报告 → 回归测试用例 → 重放执行(通过/失败) → (mock) GitHub Issue。
常用参数（`python demo.py -h` 查看全部）：
- `--smoke`：**免 API 快速自检**，跳过 LLM，用内置诊断结果仅跑重放框架 + GitHub mock，验证管道是否端到端连通（全绿退出码 0）。适合无 Key 环境或 CI。
- `--model gpt-5.6`：临时覆盖模型（等价于设置 `OPENAI_MODEL`）。
- `--data-dir DIR`：换用自己的输入目录（需含 `trajectories.jsonl` + `architecture.md` + `PRD.md`，默认 `data/`）。
- `--output FILE`：mock GitHub Issue 的落盘路径（默认 `output/github_issues.json`）。
- `--create-issue`：**经 MCP 在真实仓库创建 Issue**（需 `GITHUB_TOKEN` + `GITHUB_REPO`，见下节；缺失时自动回退 mock）。
- `--no-github`：跳过步骤 4，不生成 GitHub Issue。
### 真实运行输出（节选）
诊断阶段，Agent 定位到全部 3 个预置问题：
```
[问题 1] 未进行退款资格校验
优先级 : P0 模块: order_service PRD: R1
轨迹 : ['T-1001', 'T-1002'] 关键轮次: [3]
[问题 2] 支付重试机制未正确实现
优先级 : P0 模块: payment_service PRD: R2
[问题 3] 库存查询延迟未降级处理
优先级 : P1 模块: inventory_service PRD: R3
```
回归测试用例被重放框架**真正执行**（先复现 bug、再验证修复）：
```
(A) 对『线上未修复』系统重放 —— 期望复现 bug（FAIL）
[FAIL] RT-001 (T-1001) 工具 verify_refund_eligibility 缺失
[FAIL] RT-002 (T-1002) process_refund 调用 3 次, 失败 3 次, 末次失败
[FAIL] RT-003 (T-1003) check_stock 最大延迟 8300ms, 阈值 5000ms
(B) 对『修复后』系统重放 —— 期望修复被验证（PASS）
[PASS] RT-001 (T-1001) 工具 verify_refund_eligibility 出现
[PASS] RT-002 (T-1002) process_refund 调用 2 次, 失败 1 次, 末次成功
[PASS] RT-003 (T-1003) check_stock 最大延迟 400ms, 阈值 5000ms
小结：复现 bug 3/3 条；修复后通过 3/3 条。
```
mock GitHub Issue 打印并写入 `output/github_issues.json`，示例：
```
title : [P0][order_service] 未进行退款资格校验
labels : ['module:order_service', 'priority:critical', 'auto-diagnosis']
body : ## 问题描述 ... ## 关联回归测试用例 - RT-001 (轨迹 T-1001 第 3 轮) ...
```
### 回归测试断言 DSL（replay 框架内置）
Agent 生成的测试用例须使用以下断言之一，框架可自动求值：
- `step_present` `{tool}`：某工具必须出现（如强制前置校验）。
- `tool_succeeds` `{tool}`：某工具最终成功、且不存在"多次失败后误报成功"。
- `latency_under` `{tool, threshold_ms}`：某工具单次延迟低于阈值。
- `final_status_is` `{value}`：任务最终状态等于给定值。
### 如何适配/扩展
- **换模型**：设置 `OPENAI_MODEL`（或 `python demo.py --model <名称>`）。`diagnoser.py` 默认 `gpt-5.6-luna`，均走 JSON 模式；更强模型对复杂/隐性问题更稳。
- **换供应商**：本项目用官方 `openai` SDK，只需再设 `OPENAI_BASE_URL` 指向兼容 OpenAI 接口的服务（如 Moonshot / 火山方舟 / 本地 vLLM），配合该供应商的 `OPENAI_API_KEY` 与 `OPENAI_MODEL` 即可，无需改代码。例如：
```bash
export OPENAI_BASE_URL=https://api.moonshot.cn/v1
export OPENAI_API_KEY=your-openai-api-key # 该供应商的 Key
export OPENAI_MODEL=kimi-k3
python demo.py
```
- **换日志**：把你自己的生产轨迹按 `data/trajectories.jsonl` 的结构（`trajectory_id / task / task_input / turns[] / final_status`，`turns` 内含 `module/tool/input/output/status/latency_ms`）落盘替换即可；同时更新 `data/architecture.md` 与 `data/PRD.md` 作为诊断依据。若轨迹字段不同，`sut.py`（重放桩）与 `replay.py`（断言求值）按新字段小幅调整。
- **接入真实 GitHub MCP**：见下一节，通过 `GITHUB_TOKEN` + `GITHUB_REPO` + `--create-issue` 把 `mock=False` 接通。
### 接入真实 GitHub MCP（需 token，默认 mock）
本实验默认 mock，`--create-issue` 才会真正联网。真实创建的实现已内置在
`github_mcp._create_issues_via_mcp()`：通过 MCP 客户端（`mcp` SDK，stdio）连接官方
GitHub MCP Server，逐个调用其 `create_issue` 工具，传入 `build_issue()` 生成的
`title / body / labels / assignees`。启用步骤：
1. 准备一个 GitHub Personal Access Token（`repo` 权限），写入 `.env` 的 `GITHUB_TOKEN`；
并设置目标仓库 `GITHUB_REPO=owner/repo`。
2. 确保本机可启动官方 GitHub MCP Server。默认启动命令用官方 Docker 镜像
`ghcr.io/github/github-mcp-server`；可用 `GITHUB_MCP_COMMAND` 覆盖为任意暴露
`create_issue` 工具的 MCP Server（token 经 `GITHUB_PERSONAL_ACCESS_TOKEN` 注入其环境）。
3. 根目录 `ch5` extra 已包含 MCP SDK；上方兼容路径仍保留旧版单项目安装。
4. 运行 `python demo.py --create-issue`。缺少 `GITHUB_TOKEN` / `GITHUB_REPO` 时会打印提示并
自动回退 mock，避免误联网。
### 局限
- 被测系统 `sut.py` 是**确定性仿真**，用于让回归测试可真正重放、给出稳定的通过/失败；真实场景下重放需对接实际系统或录制/回放的依赖桩。
- 诊断质量取决于 LLM；gpt-5.6-luna 在本数据集能稳定定位全部 3 类预置问题（R1 校验缺失 / R2 重试误报 / R3 延迟未降级），步骤 1 诊断稳定；但它倾向把 R1「校验缺失」按 T-1001、T-1002 各报一条，故常输出 4 条问题（而非上文示例合并成的 3 条）。步骤 2 生成断言时，它稳定地为「支付重试」问题选用 `final_status_is:failed`（两次实跑均如此）而非上文示例的 `tool_succeeds`——因修复后的被测系统会重试成功（final_status=success），该断言在修复后重放中判 FAIL，使修复后通过数降为 3/4 而非满绿。此外它偶尔给 `step_present`/`latency_under` 的工具名加上模块前缀（如 `order_service.verify_refund_eligibility`），与重放框架按裸工具名匹配不符，会进一步压低修复后通过数（两次实跑分别得到 3/4 与 0/4）。`--smoke` 内置用例则确定性给出 3/3。
- GitHub 创建默认 mock，不联网；`--create-issue` 才经真实 MCP Server 联网创建（需 token + repo + 可用的 GitHub MCP Server）。
- 轨迹格式为简化示意，生产环境轨迹字段更丰富（token 用量、子 Agent 调用树等）。
---
## Notes / 说明
- Prefer `--smoke` without a key. / 无 Key 优先 `--smoke`。
- Commands/code/paths/env vars are identical in both language sections. / 命令、代码、路径与环境变量在中英文两侧保持一致。
chapter5/log-diagnosis/campaign.py
+660
View File
@@ -0,0 +1,660 @@
"""Acceptance campaign for Experiment 5-8 using live HTTP, an LLM, and GitHub MCP."""
from __future__ import annotations
import argparse
import hashlib
import json
import os
import re
import shutil
import socket
import subprocess
import sys
import time
import urllib.error
import urllib.request
from dataclasses import dataclass
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
from openai import OpenAI
ROOT = Path(__file__).resolve().parent
VALIDATION = ROOT / "validation"
ARCHITECTURE = """# Live diagnosis experiment architecture
The orchestrator calls a local HTTP order service. Refund flows MUST call
`verify_refund_eligibility` before `process_refund`. Inventory origin calls
have a 250 ms client deadline; on timeout the orchestrator MUST call the same
`check_stock` operation through the degraded cache route and finish normally.
Every trajectory turn records its measured HTTP latency and raw response.
"""
PRD = """# Live diagnosis experiment PRD
- R1 (P0): Every refund must call `verify_refund_eligibility` before
`process_refund`; a refund without the check is a policy violation.
- R2 (P1): `check_stock` must complete within 250 ms. On origin timeout the
orchestrator must use the degraded cache route; it must not simply fail.
- R3 (P1): Regression cases must cite the source trajectory ID and the exact
observed turn where the violation is visible.
"""
INVENTORY_DEADLINE_SECONDS = 0.250
INVENTORY_ORIGIN_HEDGE_SECONDS = 0.100
def _utc() -> str:
return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
def _write_json(path: Path, value: Any) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
text = json.dumps(value, ensure_ascii=False, indent=2) + "\n"
if re.search(r"\b(?:sk|gh[opusr])-[A-Za-z0-9_-]{12,}\b", text):
raise ValueError(f"credential-shaped value in {path}")
path.write_text(text, encoding="utf-8")
def _sha(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def _free_port() -> int:
with socket.socket() as sock:
sock.bind(("127.0.0.1", 0))
return int(sock.getsockname()[1])
@dataclass
class Backend:
provider: str
model: str
endpoint: str
client: OpenAI
receipt_checkpoint: Path | None = None
def _backend(provider: str) -> Backend:
if provider == "ark":
key = os.environ.get("ARK_API_KEY")
endpoint = os.environ.get("ARK_BASE_URL") or "https://ark.cn-beijing.volces.com/api/v3"
model = os.environ.get("ARK_MODEL") or "doubao-seed-1-6-250615"
elif provider == "moonshot":
key = os.environ.get("MOONSHOT_API_KEY") or os.environ.get("KIMI_API_KEY")
endpoint = os.environ.get("MOONSHOT_BASE_URL") or "https://api.moonshot.cn/v1"
model = os.environ.get("KIMI_MODEL") or "kimi-k3"
else:
key = os.environ.get("OPENAI_API_KEY")
endpoint = os.environ.get("OPENAI_BASE_URL") or "https://api.openai.com/v1"
model = os.environ.get("OPENAI_MODEL") or "gpt-5.6-luna"
if not key:
raise RuntimeError(f"missing credential for {provider}")
return Backend(
provider,
model,
endpoint,
OpenAI(api_key=key, base_url=endpoint, timeout=180, max_retries=0),
)
def _json_object(text: str) -> dict[str, Any]:
text = text.strip()
if text.startswith("```"):
text = text.split("\n", 1)[1].rsplit("```", 1)[0]
value = json.loads(text)
if not isinstance(value, dict):
raise ValueError("response is not an object")
return value
def _llm_call(
backend: Backend,
messages: list[dict[str, str]],
purpose: str,
receipts: list[dict[str, Any]],
) -> dict[str, Any]:
started = time.perf_counter()
request = {
"model": backend.model,
"messages": messages,
"response_format": {"type": "json_object"},
"temperature": 0,
}
try:
response = backend.client.chat.completions.create(**request)
except Exception as exc:
receipts.append(
{
"purpose": purpose,
"provider": backend.provider,
"endpoint": backend.endpoint,
"request": request,
"latency_s": round(time.perf_counter() - started, 3),
"error": f"{type(exc).__name__}: {exc}",
}
)
if backend.receipt_checkpoint is not None:
_write_json(backend.receipt_checkpoint, {"calls": receipts})
raise
usage = response.usage
content = response.choices[0].message.content or ""
receipt = {
"purpose": purpose,
"provider": backend.provider,
"endpoint": backend.endpoint,
"request": request,
"latency_s": round(time.perf_counter() - started, 3),
"response": {
"id": response.id,
"model": response.model,
"finish_reason": response.choices[0].finish_reason,
"content": content,
},
"usage": {
"prompt_tokens": getattr(usage, "prompt_tokens", None),
"completion_tokens": getattr(usage, "completion_tokens", None),
"total_tokens": getattr(usage, "total_tokens", None),
},
}
receipts.append(receipt)
if backend.receipt_checkpoint is not None:
_write_json(backend.receipt_checkpoint, {"calls": receipts})
return _json_object(content)
def _http_call(
base: str,
method: str,
path: str,
*,
body: dict[str, Any] | None = None,
timeout: float = 2.0,
) -> dict[str, Any]:
raw = None if body is None else json.dumps(body).encode("utf-8")
request = urllib.request.Request(
base + path,
data=raw,
method=method,
headers={"content-type": "application/json"},
)
started = time.perf_counter()
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
response_raw = response.read()
elapsed = round((time.perf_counter() - started) * 1000, 3)
return {
"method": method,
"path": path,
"request": body,
"http_status": response.status,
"status": "success",
"latency_ms": elapsed,
"response": json.loads(response_raw),
}
except Exception as exc:
return {
"method": method,
"path": path,
"request": body,
"http_status": getattr(exc, "code", None),
"status": "error",
"latency_ms": round((time.perf_counter() - started) * 1000, 3),
"error": f"{type(exc).__name__}: {exc}",
}
def _trajectory(
base: str,
task_input: dict[str, Any],
source_id: str,
*,
inject_regressions: bool = False,
) -> dict[str, Any]:
"""Run the HTTP orchestrator and return its measured trajectory.
Correct behavior is the default. The acceptance campaign can explicitly
inject the historical issue #502 behavior to prove generated regression
tests fail before the fix and pass against the production policy.
"""
turns: list[dict[str, Any]] = []
def call(tool: str, method: str, path: str, **kwargs: Any) -> dict[str, Any]:
observed = _http_call(base, method, path, **kwargs)
turns.append({"index": len(turns), "role": "tool", "module": "http_order_system", "tool": tool, **observed})
return observed
intent = task_input["intent"]
order_id = task_input["order_id"]
turns.append({"index": 0, "role": "user", "content": json.dumps(task_input, ensure_ascii=False)})
call("query_order", "GET", f"/orders/{order_id}")
final_status = "success"
if intent == "refund":
if inject_regressions:
call("process_refund", "POST", "/refund/process", body={"order_id": order_id})
else:
eligibility = call(
"verify_refund_eligibility",
"POST",
"/refund/eligibility",
body={"order_id": order_id},
)
eligibility_response = eligibility.get("response")
eligible = (
eligibility["status"] == "success"
and isinstance(eligibility_response, dict)
and eligibility_response.get("eligible") is True
)
if eligible:
refund = call(
"process_refund",
"POST",
"/refund/process",
body={"order_id": order_id},
)
if refund["status"] != "success":
final_status = "failed"
else:
final_status = "failed"
elif intent == "order_status":
# Hedge the origin request early enough to leave time for the cache
# fallback inside the end-to-end 250 ms inventory deadline.
inventory_started = time.perf_counter()
origin_timeout = (
INVENTORY_DEADLINE_SECONDS + 0.100
if inject_regressions
else INVENTORY_ORIGIN_HEDGE_SECONDS
)
origin = call(
"check_stock", "GET", f"/inventory/{task_input['sku']}", timeout=origin_timeout
)
if origin["status"] == "error":
if inject_regressions:
final_status = "failed"
else:
elapsed = time.perf_counter() - inventory_started
remaining = max(0.001, INVENTORY_DEADLINE_SECONDS - elapsed)
degraded = call(
"check_stock",
"GET",
f"/inventory/{task_input['sku']}?degraded=1",
timeout=remaining,
)
if degraded["status"] != "success":
final_status = "failed"
call("notify_user", "POST", "/notifications", body={"status": final_status})
implementation = "buggy" if inject_regressions else "fixed"
return {
"trajectory_id": f"{source_id}::{implementation}",
"source_trajectory_id": source_id,
"implementation": implementation,
"task_input": task_input,
"final_status": final_status,
"turns": turns,
}
def _tool_turns(trajectory: dict[str, Any], tool: str) -> list[dict[str, Any]]:
return [turn for turn in trajectory["turns"] if turn.get("tool") == tool]
def _evaluate(assertion: dict[str, Any], trajectory: dict[str, Any]) -> tuple[bool, str]:
kind = assertion.get("type")
params = assertion.get("params") or {}
if kind == "step_present":
tool = str(params.get("tool") or "")
count = len(_tool_turns(trajectory, tool))
return count > 0, f"{tool} calls={count}"
if kind == "latency_under":
tool = str(params.get("tool") or "")
threshold = float(params.get("threshold_ms"))
calls = _tool_turns(trajectory, tool)
# A timed-out attempt is part of the operation and therefore counts.
worst = max((float(turn["latency_ms"]) for turn in calls), default=float("inf"))
return worst < threshold, f"{tool} max_latency_ms={worst:.3f}, threshold_ms={threshold:.3f}"
if kind == "final_status_is":
wanted = str(params.get("value") or "")
actual = str(trajectory.get("final_status") or "")
return actual == wanted, f"final_status={actual}, expected={wanted}"
return False, f"unsupported assertion type: {kind}"
def _validate_diagnosis(payload: dict[str, Any], sources: dict[str, dict[str, Any]]) -> list[dict[str, Any]]:
problems = payload.get("problems")
if not isinstance(problems, list) or len(problems) < 2:
raise ValueError("diagnosis must contain at least two evidence-backed problems")
refs: set[str] = set()
for problem in problems:
if not isinstance(problem, dict):
raise ValueError("problem is not an object")
refs.add(str(problem.get("prd_ref")))
tids = problem.get("trajectory_ids")
turns = problem.get("focus_turns")
if not isinstance(tids, list) or not tids or not all(tid in sources for tid in tids):
raise ValueError(
"problem contains an unknown trajectory reference; trajectory_ids must use only "
+ json.dumps(sorted(sources))
+ " (the source_trajectory_id values, without the ::buggy suffix)"
)
if not isinstance(turns, list) or not turns or not all(isinstance(value, int) for value in turns):
raise ValueError("problem lacks concrete focus turns")
if not all(
any(0 <= value < len(sources[tid]["turns"]) for tid in tids)
for value in turns
):
raise ValueError("problem focus_turns contain indexes absent from every cited trajectory")
if not {"R1", "R2"}.issubset(refs):
raise ValueError(f"diagnosis did not cover both observed PRD violations: {sorted(refs)}")
return problems
def _validate_tests(payload: dict[str, Any], sources: dict[str, dict[str, Any]]) -> list[dict[str, Any]]:
tests = payload.get("test_cases")
if not isinstance(tests, list) or len(tests) < 2:
raise ValueError("at least two regression tests are required")
kinds: set[str] = set()
for test in tests:
if not isinstance(test, dict) or test.get("trajectory_id") not in sources:
raise ValueError("test has an unknown trajectory ID")
focus = test.get("focus_turn")
if not isinstance(focus, int) or not (0 <= focus < len(sources[test["trajectory_id"]]["turns"])):
raise ValueError("test focus_turn is not an observed source turn")
assertion = test.get("assertion")
if not isinstance(assertion, dict) or assertion.get("type") not in {"step_present", "latency_under", "final_status_is"}:
raise ValueError("test assertion is not executable by the frozen DSL")
kinds.add(assertion["type"])
if "step_present" not in kinds or "latency_under" not in kinds:
raise ValueError("tests must cover both the missing prerequisite and latency violations")
return tests
def _mcp_create_issue(title: str, body: str, repo: str, token: str) -> tuple[dict[str, Any], dict[str, Any], str]:
import asyncio
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
owner, sep, name = repo.partition("/")
if not sep:
raise ValueError("repository must be owner/name")
request = {
"method": "create",
"owner": owner,
"repo": name,
"title": title,
"body": body,
}
params = StdioServerParameters(
command="github-mcp-server",
args=["stdio", "--toolsets=issues"],
env={**os.environ, "GITHUB_PERSONAL_ACCESS_TOKEN": token},
)
async def run() -> tuple[dict[str, Any], str]:
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
tools = await session.list_tools()
names = [tool.name for tool in tools.tools]
if "issue_write" not in names:
raise RuntimeError("official GitHub MCP server did not expose issue_write")
result = await session.call_tool("issue_write", request)
text = "".join(getattr(item, "text", "") for item in result.content)
return {"is_error": bool(result.isError), "content": text}, "github-mcp-server stdio"
response, transport = asyncio.run(run())
if response["is_error"]:
raise RuntimeError(f"GitHub MCP create_issue failed: {response['content']}")
match = re.search(r"https://github\.com/[^\s\"']+/issues/\d+", response["content"])
if not match:
raise RuntimeError(f"GitHub MCP response lacks issue URL: {response['content']}")
return request, response, match.group(0)
def run(provider: str, run_id: str, repo: str) -> dict[str, Any]:
run_dir = VALIDATION / "runs" / run_id
if run_dir.exists():
raise FileExistsError(run_dir)
run_dir.mkdir(parents=True)
(run_dir / "architecture.md").write_text(ARCHITECTURE, encoding="utf-8")
(run_dir / "PRD.md").write_text(PRD, encoding="utf-8")
shutil.copy2(ROOT / "http_service.py", run_dir / "http_service.py")
port = _free_port()
service_log = (run_dir / "service.stdout.jsonl").open("w", encoding="utf-8")
service_err = (run_dir / "service.stderr.log").open("w", encoding="utf-8")
process = subprocess.Popen(
[sys.executable, "-u", str(ROOT / "http_service.py"), "--port", str(port)],
stdout=service_log,
stderr=service_err,
text=True,
)
base = f"http://127.0.0.1:{port}"
try:
for _ in range(50):
health = _http_call(base, "GET", "/health", timeout=0.2)
if health["status"] == "success":
break
time.sleep(0.05)
else:
raise RuntimeError("local HTTP service did not become healthy")
source_tasks = {
"HTTP-RF-001": {"intent": "refund", "order_id": "ORD-58-A"},
"HTTP-INV-001": {"intent": "order_status", "order_id": "ORD-58-B", "sku": "SKU-42"},
}
sources = {
tid: _trajectory(base, task, tid, inject_regressions=True)
for tid, task in source_tasks.items()
}
with (run_dir / "production_trajectories.jsonl").open("w", encoding="utf-8") as handle:
for trajectory in sources.values():
handle.write(json.dumps(trajectory, ensure_ascii=False) + "\n")
backend = _backend(provider)
receipts: list[dict[str, Any]] = []
backend.receipt_checkpoint = run_dir / "provider_receipts.checkpoint.json"
diagnosis_prompt = f"""Architecture:\n{ARCHITECTURE}\n\nPRD:\n{PRD}\n\nObserved production trajectories:\n{json.dumps(list(sources.values()), ensure_ascii=False, indent=2)}\n\nDiagnose only evidenced violations. Return JSON {{\"problems\":[...]}}. Each problem must contain title, priority, module, description, suggestion, prd_ref, trajectory_ids, focus_turns, and suggested_assignee. focus_turns must cite exact integer indexes in the supplied trajectories.
The ONLY allowed trajectory_ids strings are the exact source_trajectory_id values {json.dumps(sorted(sources))}. Do not copy the implementation-specific trajectory_id values ending in ::buggy. For R1 cite HTTP-RF-001 and exact visible turn indexes; for R2 cite HTTP-INV-001 and exact visible turn indexes."""
problems: list[dict[str, Any]] | None = None
diagnosis_error = ""
for attempt in range(1, 4):
payload = _llm_call(
backend,
[
{"role": "system", "content": "You are a production Agent diagnostician. Return one JSON object only and never invent evidence."},
{"role": "user", "content": diagnosis_prompt + (f"\n\nPrior validation error: {diagnosis_error}" if diagnosis_error else "")},
],
f"diagnosis_attempt_{attempt}",
receipts,
)
try:
problems = _validate_diagnosis(payload, sources)
break
except ValueError as exc:
diagnosis_error = str(exc)
if problems is None:
raise RuntimeError(f"model diagnosis never passed evidence gates: {diagnosis_error}")
_write_json(run_dir / "diagnosis.json", {"problems": problems})
test_prompt = f"""PRD:\n{PRD}\n\nModel-diagnosed problems:\n{json.dumps(problems, ensure_ascii=False, indent=2)}\n\nCreate executable regression tests. Return JSON {{\"test_cases\":[...]}}. Each test needs test_id, trajectory_id, focus_turn, description, and assertion. Allowed assertions only:\n- {{\"type\":\"step_present\",\"params\":{{\"tool\":\"...\"}}}}\n- {{\"type\":\"latency_under\",\"params\":{{\"tool\":\"...\",\"threshold_ms\":250}}}}\n- {{\"type\":\"final_status_is\",\"params\":{{\"value\":\"success\"}}}}\nTests must cite exact source trajectory IDs and turns. Cover both R1 and R2, expressing the correct fixed behavior.
The ONLY allowed trajectory_id strings are {json.dumps(sorted(sources))}; use HTTP-RF-001 for the refund prerequisite assertion and HTTP-INV-001 for the inventory latency assertion. Do not append ::buggy."""
tests: list[dict[str, Any]] | None = None
replay_records: list[dict[str, Any]] = []
feedback = ""
for attempt in range(1, 4):
payload = _llm_call(
backend,
[
{"role": "system", "content": "You are a regression-test engineer. Return one JSON object only."},
{"role": "user", "content": test_prompt + (f"\n\nPrior executable validation feedback:\n{feedback}" if feedback else "")},
],
f"regression_generation_attempt_{attempt}",
receipts,
)
try:
candidate = _validate_tests(payload, sources)
observed: list[dict[str, Any]] = []
for test in candidate:
source_id = test["trajectory_id"]
task = sources[source_id]["task_input"]
buggy_traj = _trajectory(
base, task, source_id, inject_regressions=True
)
fixed_traj = _trajectory(base, task, source_id)
buggy_passed, buggy_detail = _evaluate(test["assertion"], buggy_traj)
fixed_passed, fixed_detail = _evaluate(test["assertion"], fixed_traj)
observed.append(
{
"test": test,
"buggy": {"passed": buggy_passed, "detail": buggy_detail, "trajectory": buggy_traj},
"fixed": {"passed": fixed_passed, "detail": fixed_detail, "trajectory": fixed_traj},
}
)
replay_records = observed
if not all(not item["buggy"]["passed"] and item["fixed"]["passed"] for item in observed):
raise ValueError(
"every generated test must reproduce on buggy and pass on fixed: "
+ json.dumps(
[
{
"test_id": item["test"].get("test_id"),
"buggy": item["buggy"]["detail"],
"buggy_passed": item["buggy"]["passed"],
"fixed": item["fixed"]["detail"],
"fixed_passed": item["fixed"]["passed"],
}
for item in observed
],
ensure_ascii=False,
)
)
tests = candidate
break
except ValueError as exc:
feedback = str(exc)
if tests is None:
raise RuntimeError(f"model-generated tests never passed live replay: {feedback}")
_write_json(run_dir / "regression_tests.json", {"test_cases": tests})
_write_json(run_dir / "live_replays.json", {"results": replay_records})
_write_json(run_dir / "provider_receipts.json", {"calls": receipts})
issue_title = f"[Experiment 5-8][auto-diagnosis] Live HTTP regression findings ({run_id})"
issue_body = (
"This issue was created automatically by the Chapter 5 Experiment 5-8 acceptance campaign.\n\n"
"## Evidence-backed diagnosis\n\n"
+ "\n".join(
f"- **{p['priority']} {p['title']}** ({p['prd_ref']}): {p['description']} "
f"Trajectories: {', '.join(p['trajectory_ids'])}; turns: {p['focus_turns']}."
for p in problems
)
+ "\n\n## Generated regression tests\n\n"
+ "\n".join(
f"- `{t['test_id']}` cites `{t['trajectory_id']}` turn {t['focus_turn']}: "
f"`{json.dumps(t['assertion'], ensure_ascii=False)}`"
for t in tests
)
+ "\n\nThe campaign replayed each test against the live buggy and fixed HTTP orchestrators: "
"all tests failed on buggy behavior and passed on fixed behavior."
)
token = subprocess.run(
["gh", "auth", "token"], check=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True
).stdout.strip()
mcp_request, mcp_response, issue_url = _mcp_create_issue(issue_title, issue_body, repo, token)
_write_json(
run_dir / "github_mcp_receipt.json",
{
"server": "official github/github-mcp-server",
"transport": "stdio",
"tool": "issue_write(method=create)",
"request": mcp_request,
"response": mcp_response,
"issue_url": issue_url,
"credential_free": True,
},
)
gates = {
"real_local_http_trajectories": len(sources) == 2 and all(t["turns"] for t in sources.values()),
"measured_latency_and_raw_http": all(
all("latency_ms" in turn and ("response" in turn or "error" in turn) for turn in trajectory["turns"] if turn.get("role") == "tool")
for trajectory in sources.values()
),
"live_model_diagnosis": bool(problems) and bool(receipts),
"diagnosis_references_trajectories_and_turns": all(p.get("trajectory_ids") and p.get("focus_turns") for p in problems),
"live_model_generated_executable_tests": bool(tests),
"buggy_failures_reproduced": all(not item["buggy"]["passed"] for item in replay_records),
"fixed_system_passes": all(item["fixed"]["passed"] for item in replay_records),
"official_github_mcp_issue_created": issue_url.startswith(f"https://github.com/{repo}/issues/"),
"raw_provider_receipts_complete": all(
receipt.get("response")
and receipt.get("usage", {}).get("prompt_tokens") is not None
and receipt.get("usage", {}).get("completion_tokens") is not None
and receipt.get("latency_s") is not None
for receipt in receipts
),
}
official_complete = all(gates.values())
process.terminate()
process.wait(timeout=5)
service_log.close()
service_err.close()
artifacts = {
str(path.relative_to(run_dir)): {"sha256": _sha(path), "bytes": path.stat().st_size}
for path in sorted(run_dir.rglob("*"))
if path.is_file()
}
manifest = {
"schema_version": "1.0",
"experiment": "5-8",
"run_id": run_id,
"generated_at_utc": _utc(),
"provider": backend.provider,
"model": backend.model,
"service": {"kind": "real local HTTP subprocess", "base_url": base, "pid": process.pid},
"source_trajectory_ids": sorted(sources),
"model_call_count": len(receipts),
"github_issue_url": issue_url,
"gates": gates,
"artifacts": artifacts,
"official_complete": official_complete,
}
_write_json(run_dir / "manifest.json", manifest)
if not official_complete:
raise RuntimeError("Experiment 5-8 gates failed")
latest = {
"experiment": "5-8",
"run_id": run_id,
"manifest": str((run_dir / "manifest.json").relative_to(ROOT)),
"manifest_sha256": _sha(run_dir / "manifest.json"),
"official_complete": True,
}
_write_json(VALIDATION / "latest.json", latest)
return manifest
finally:
if process.poll() is None:
process.terminate()
try:
process.wait(timeout=5)
except subprocess.TimeoutExpired:
process.kill()
if not service_log.closed:
service_log.close()
if not service_err.closed:
service_err.close()
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--provider", choices=("ark", "moonshot", "openai"), default="ark")
parser.add_argument("--repo", default="bojieli/ai-agent-book")
parser.add_argument("--run-id", default=f"exp5-8-live-http-mcp-{datetime.now().strftime('%Y%m%d-%H%M%S')}")
args = parser.parse_args()
print(json.dumps(run(args.provider, args.run_id, args.repo), ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
chapter5/log-diagnosis/data/PRD.md
+15
View File
@@ -0,0 +1,15 @@
# 产品需求文档（PRD，精简版）—— 订单退款 Agent
## R1 退款合规校验（P0）
发起任何退款（`process_refund`）之前，**必须**先调用 `verify_refund_eligibility` 完成退款资格校验。
未校验直接退款属于严重合规/资损风险，禁止出现。
## R2 支付重试与上报（P0）
`process_refund` 调用第三方支付网关。网关偶发失败时，系统应带退避重试；
若最终仍失败，必须将任务标记为 `failed` 并通知用户，**不得**在多次失败后静默结束或误报成功。
## R3 库存查询延迟（P1）
`check_stock` 单次调用延迟必须小于 **5000ms**。超过阈值应触发降级（返回缓存或"库存查询繁忙"提示），不得让用户长时间等待。
## R4 结果通知（P2）
任务结束（成功或失败）都应通过 `notification_service.notify_user` 通知用户最终结果。
chapter5/log-diagnosis/data/architecture.md
+28
View File
@@ -0,0 +1,28 @@
# 订单退款 Agent 系统架构
本系统是一个电商客服场景下的多模块 Agent，负责处理用户的订单查询与退款请求。
## 模块划分
- **intent_parser（意图识别）**：解析用户自然语言，判断意图（`refund` / `order_status` / `chitchat`），抽取订单号等实体。
- **order_service（订单服务）**：负责订单相关的读写。
- 工具 `query_order(order_id)`：查询订单状态与金额。
- 工具 `verify_refund_eligibility(order_id)`：校验订单是否满足退款条件（是否已支付、是否在退款时限内、是否已退过款）。**退款流程中此步骤为强制前置校验。**
- **payment_service（支付服务）**：对接第三方支付网关。
- 工具 `process_refund(order_id, amount)`：发起退款。第三方网关偶发不稳定，要求实现**重试 + 退避**并在最终失败时上报，不得静默丢弃。
- **inventory_service（库存服务）**：查询与回补库存。
- 工具 `check_stock(sku)`：查询库存。**要求单次调用延迟不超过 5000ms**，超时应走降级路径而非阻塞主流程。
- **notification_service（通知服务）**：向用户发送结果通知。
## 关键调用链（退款）
```
user -> intent_parser -> order_service.query_order
-> order_service.verify_refund_eligibility # 强制前置校验
-> payment_service.process_refund # 需重试+退避
-> notification_service.notify_user
```
## 可观测性
系统为每次任务落一条 trajectory（轨迹）日志，记录每一交互轮次（turn）的模块、工具、输入输出、状态与延迟（latency_ms）。轨迹用于线上问题诊断与回归测试重放。
chapter5/log-diagnosis/data/trajectories.jsonl
+4
View File
@@ -0,0 +1,4 @@
{"trajectory_id": "T-1001", "task": "用户要求对订单 A1001 退款", "task_input": {"intent": "refund", "order_id": "A1001", "order_status": "paid", "payment_flaky": false, "slow_inventory": false}, "final_status": "success", "turns": [{"index": 0, "role": "user", "content": "我要退款，订单号 A1001"}, {"index": 1, "role": "assistant", "module": "intent_parser", "content": "意图=refund, order_id=A1001"}, {"index": 2, "role": "tool", "module": "order_service", "tool": "query_order", "input": {"order_id": "A1001"}, "output": {"status": "paid", "amount": 199}, "status": "success", "latency_ms": 210}, {"index": 3, "role": "assistant", "module": "order_service", "content": "订单已支付，直接发起退款", "note": "未调用 verify_refund_eligibility"}, {"index": 4, "role": "tool", "module": "payment_service", "tool": "process_refund", "input": {"order_id": "A1001", "amount": 199}, "output": {"refund_id": "R-9001"}, "status": "success", "latency_ms": 540}, {"index": 5, "role": "tool", "module": "notification_service", "tool": "notify_user", "input": {"msg": "退款成功"}, "output": {"sent": true}, "status": "success", "latency_ms": 60}]}
{"trajectory_id": "T-1002", "task": "用户要求对订单 A1002 退款", "task_input": {"intent": "refund", "order_id": "A1002", "order_status": "paid", "payment_flaky": true, "slow_inventory": false}, "final_status": "success", "turns": [{"index": 0, "role": "user", "content": "订单 A1002 退款"}, {"index": 1, "role": "assistant", "module": "intent_parser", "content": "意图=refund, order_id=A1002"}, {"index": 2, "role": "tool", "module": "order_service", "tool": "query_order", "input": {"order_id": "A1002"}, "output": {"status": "paid", "amount": 88}, "status": "success", "latency_ms": 230}, {"index": 3, "role": "assistant", "module": "order_service", "content": "订单已支付，直接发起退款", "note": "未调用 verify_refund_eligibility"}, {"index": 4, "role": "tool", "module": "payment_service", "tool": "process_refund", "input": {"order_id": "A1002", "amount": 88}, "output": {"error": "gateway_timeout"}, "status": "error", "latency_ms": 3010}, {"index": 5, "role": "tool", "module": "payment_service", "tool": "process_refund", "input": {"order_id": "A1002", "amount": 88}, "output": {"error": "gateway_timeout"}, "status": "error", "latency_ms": 3005}, {"index": 6, "role": "tool", "module": "payment_service", "tool": "process_refund", "input": {"order_id": "A1002", "amount": 88}, "output": {"error": "gateway_timeout"}, "status": "error", "latency_ms": 3020}, {"index": 7, "role": "assistant", "module": "payment_service", "content": "重试多次失败，按成功结束", "note": "无退避；最终误报成功"}, {"index": 8, "role": "tool", "module": "notification_service", "tool": "notify_user", "input": {"msg": "退款成功"}, "output": {"sent": true}, "status": "success", "latency_ms": 55}]}
{"trajectory_id": "T-1003", "task": "用户查询订单 A1003 的库存/发货状态", "task_input": {"intent": "order_status", "order_id": "A1003", "sku": "SKU-77", "payment_flaky": false, "slow_inventory": true}, "final_status": "success", "turns": [{"index": 0, "role": "user", "content": "A1003 还有货吗，什么时候发"}, {"index": 1, "role": "assistant", "module": "intent_parser", "content": "意图=order_status, order_id=A1003"}, {"index": 2, "role": "tool", "module": "order_service", "tool": "query_order", "input": {"order_id": "A1003"}, "output": {"status": "paid", "sku": "SKU-77"}, "status": "success", "latency_ms": 240}, {"index": 3, "role": "tool", "module": "inventory_service", "tool": "check_stock", "input": {"sku": "SKU-77"}, "output": {"stock": 12}, "status": "success", "latency_ms": 8300, "note": "延迟超过 5000ms 阈值，未降级"}, {"index": 4, "role": "tool", "module": "notification_service", "tool": "notify_user", "input": {"msg": "有货，1天内发货"}, "output": {"sent": true}, "status": "success", "latency_ms": 70}]}
{"trajectory_id": "T-1004", "task": "用户查询订单 A1004 状态（正常轨迹，对照组）", "task_input": {"intent": "order_status", "order_id": "A1004", "sku": "SKU-12", "payment_flaky": false, "slow_inventory": false}, "final_status": "success", "turns": [{"index": 0, "role": "user", "content": "A1004 到哪了"}, {"index": 1, "role": "assistant", "module": "intent_parser", "content": "意图=order_status, order_id=A1004"}, {"index": 2, "role": "tool", "module": "order_service", "tool": "query_order", "input": {"order_id": "A1004"}, "output": {"status": "shipped", "sku": "SKU-12"}, "status": "success", "latency_ms": 200}, {"index": 3, "role": "tool", "module": "inventory_service", "tool": "check_stock", "input": {"sku": "SKU-12"}, "output": {"stock": 5}, "status": "success", "latency_ms": 300}, {"index": 4, "role": "tool", "module": "notification_service", "tool": "notify_user", "input": {"msg": "已发货"}, "output": {"sent": true}, "status": "success", "latency_ms": 65}]}
chapter5/log-diagnosis/demo.py
+218
View File
@@ -0,0 +1,218 @@
"""
demo.py —— 实验 5-8：生产日志的智能诊断系统（全流程演示）
流水线：
读轨迹集合 + 架构 + PRD
-> [LLM] 诊断：定位问题、结构化报告(优先级/模块/描述/建议)
-> [LLM] 生成回归测试用例(引用轨迹ID+交互轮次)
-> 重放框架真正执行：先复现 bug(FAIL)，再验证修复(PASS)
-> (mock) 通过 MCP 对接 GitHub 创建 Issue
运行：
cp env.example .env && 填入 OPENAI_API_KEY
python demo.py # 完整流程（两次真实 LLM 调用，GitHub 步骤默认 mock）
python demo.py --smoke # 快速自检：跳过 LLM，用内置用例仅跑重放+GitHub mock
python demo.py --model gpt-5.6 # 临时切换模型
python demo.py --create-issue # 真正经 MCP 创建 GitHub Issue（需 GITHUB_TOKEN+GITHUB_REPO）
python demo.py -h # 查看全部参数
换供应商/模型：设置 OPENAI_BASE_URL + OPENAI_MODEL（见 README『如何适配/扩展』）。
"""
import argparse
import json
import os
import sys
try:
from dotenv import load_dotenv
load_dotenv()
except Exception:
pass
from diagnoser import Diagnoser
import replay
import github_mcp
HERE = os.path.dirname(os.path.abspath(__file__))
DATA = os.path.join(HERE, "data")
DEFAULT_OUTPUT = os.path.join(HERE, "output", "github_issues.json")
# --smoke 自检用的内置样例：与 LLM 在本数据集上的稳定产物一致，
# 使得无需联网/无需 API Key 即可验证 重放框架 + GitHub mock 的端到端管道。
_CANNED_PROBLEMS = [
{"title": "未进行退款资格校验", "priority": "P0", "module": "order_service",
"description": "退款前缺失强制的 verify_refund_eligibility 校验。", "prd_ref": "R1",
"trajectory_ids": ["T-1001", "T-1002"], "focus_turns": [3]},
{"title": "支付重试机制未正确实现", "priority": "P0", "module": "payment_service",
"description": "process_refund 反复失败、无退避、且最终误报成功。", "prd_ref": "R2",
"trajectory_ids": ["T-1002"], "focus_turns": [7]},
{"title": "库存查询延迟未降级处理", "priority": "P1", "module": "inventory_service",
"description": "check_stock 延迟 8300ms 超时未降级。", "prd_ref": "R3",
"trajectory_ids": ["T-1003"], "focus_turns": [3]},
]
_CANNED_TEST_CASES = [
{"test_id": "RT-001", "trajectory_id": "T-1001", "focus_turn": 3,
"description": "退款前必须先做资格校验",
"assertion": {"type": "step_present", "params": {"tool": "verify_refund_eligibility"}}},
{"test_id": "RT-002", "trajectory_id": "T-1002", "focus_turn": 7,
"description": "process_refund 应最终成功且无『多次失败后误报成功』",
"assertion": {"type": "tool_succeeds", "params": {"tool": "process_refund"}}},
{"test_id": "RT-003", "trajectory_id": "T-1003", "focus_turn": 3,
"description": "check_stock 延迟应低于 5000ms",
"assertion": {"type": "latency_under", "params": {"tool": "check_stock", "threshold_ms": 5000}}},
]
def _read(data_dir, name):
with open(os.path.join(data_dir, name), "r", encoding="utf-8") as f:
return f.read()
def _traj_path(data_dir):
return os.path.join(data_dir, "trajectories.jsonl")
def _hr(title):
print("\n" + "=" * 70)
print(title)
print("=" * 70)
def _replay_and_issues(problems, test_cases, do_github=True,
traj_path=None, out_path=DEFAULT_OUTPUT, create_issue=False):
"""步骤 3/4：对同一输入重放被测系统并断言，再生成 GitHub Issue（默认 mock）。
对 fixed=False / fixed=True 各重放一次，演示同一条回归用例的
『失败(复现bug)』与『通过(验证修复)』。返回 (复现数, 验证数)。
"""
traj_path = traj_path or replay._DATA
_hr("步骤 3｜重放框架真正执行测试用例")
print("(A) 对『线上未修复』系统重放 —— 期望复现 bug（FAIL）")
buggy = replay.run_suite(test_cases, fixed=False, path=traj_path)
for r in buggy:
flag = "PASS" if r["passed"] else "FAIL"
print(f" [{flag}] {r['test_id']} ({r.get('trajectory_id')}) {r['detail']}")
print("\n(B) 对『修复后』系统重放 —— 期望修复被验证（PASS）")
fixed = replay.run_suite(test_cases, fixed=True, path=traj_path)
for r in fixed:
flag = "PASS" if r["passed"] else "FAIL"
print(f" [{flag}] {r['test_id']} ({r.get('trajectory_id')}) {r['detail']}")
reproduced = sum(1 for r in buggy if not r["passed"])
verified = sum(1 for r in fixed if r["passed"])
print(f"\n 小结：复现 bug {reproduced}/{len(buggy)} 条；修复后通过 {verified}/{len(fixed)} 条。")
if do_github:
token, repo = os.getenv("GITHUB_TOKEN"), os.getenv("GITHUB_REPO")
if create_issue and token and repo:
_hr(f"步骤 4｜通过 MCP 对接 GitHub 在 {repo} 真实创建 Issue")
github_mcp.create_issues(problems, test_cases, mock=False,
out_path=out_path, repo=repo, token=token)
else:
if create_issue:
print("\n[提示] --create-issue 需要 GITHUB_TOKEN 与 GITHUB_REPO(owner/repo)，"
"当前缺失，已回退到 mock。")
_hr("步骤 4｜通过 MCP 对接 GitHub 创建 Issue（mock，不联网）")
github_mcp.create_issues(problems, test_cases, mock=True, out_path=out_path)
return reproduced, verified
def run_smoke(data_dir=DATA, out_path=DEFAULT_OUTPUT):
"""快速自检：不调用 LLM，用内置样例仅跑 重放框架 + GitHub mock 的端到端管道。
退出码：管道全绿(复现全部 + 验证全部)返回 0，否则返回 3。
"""
_hr("自检模式（--smoke）：跳过 LLM，用内置诊断结果验证重放+GitHub mock 管道")
reproduced, verified = _replay_and_issues(
_CANNED_PROBLEMS, _CANNED_TEST_CASES,
traj_path=_traj_path(data_dir), out_path=out_path)
n = len(_CANNED_TEST_CASES)
ok = reproduced == n and verified == n
print(f"\n自检结果：{'OK' if ok else 'FAILED'}（复现 {reproduced}/{n}，验证 {verified}/{n}）")
return 0 if ok else 3
def run_full(model=None, do_github=True, data_dir=DATA,
out_path=DEFAULT_OUTPUT, create_issue=False):
"""完整流程：真实调用 OpenAI 诊断并生成回归用例，再重放执行。"""
if not (os.getenv("OPENAI_API_KEY") or os.getenv("OPENROUTER_API_KEY")):
print("错误：未设置 OPENAI_API_KEY（或 OPENROUTER_API_KEY 兜底），请 cp env.example .env 后填入"
"（或用 python demo.py --smoke 免 API 自检）。")
sys.exit(1)
# ---------- 0. 读取输入 ----------
architecture = _read(data_dir, "architecture.md")
prd = _read(data_dir, "PRD.md")
trajectories = list(replay.load_trajectories(_traj_path(data_dir)).values())
_hr(f"步骤 0｜读取输入：{len(trajectories)} 条生产轨迹 + 架构文档 + PRD")
for t in trajectories:
print(f" - {t['trajectory_id']}: {t['task']}（{len(t['turns'])} 轮）")
agent = Diagnoser(model=model) if model else Diagnoser()
print(f" 使用模型：{agent.model}")
# ---------- 1. 诊断：定位问题 ----------
_hr("步骤 1｜Agent 诊断（真实调用 OpenAI）：定位问题并生成结构化报告")
problems = agent.diagnose(architecture, prd, trajectories)
if not problems:
print("未诊断出问题（异常）。")
sys.exit(2)
for i, p in enumerate(problems, 1):
print(f"\n[问题 {i}] {p.get('title', '')}")
print(f" 优先级 : {p.get('priority')} 模块: {p.get('module')} PRD: {p.get('prd_ref')}")
print(f" 轨迹 : {p.get('trajectory_ids')} 关键轮次: {p.get('focus_turns')}")
print(f" 描述 : {p.get('description')}")
print(f" 建议 : {p.get('suggestion')}")
# ---------- 2. 生成回归测试用例 ----------
_hr("步骤 2｜Agent 生成回归测试用例（真实调用 OpenAI）：引用轨迹ID + 交互轮次")
test_cases = agent.gen_test_cases(problems)
for tc in test_cases:
print(f" {tc.get('test_id')} 轨迹={tc.get('trajectory_id')} "
f"轮次={tc.get('focus_turn')} 断言={json.dumps(tc.get('assertion'), ensure_ascii=False)}")
print(f" 说明: {tc.get('description')}")
# ---------- 3/4. 重放执行 + GitHub Issue（默认 mock） ----------
_replay_and_issues(problems, test_cases, do_github=do_github,
traj_path=_traj_path(data_dir), out_path=out_path,
create_issue=create_issue)
_hr("完成｜读轨迹 -> 诊断报告 -> 回归测试用例 -> (mock) GitHub Issue 全流程跑通")
def main():
parser = argparse.ArgumentParser(
description="实验 5-8：生产日志的智能诊断系统（读轨迹->诊断->回归测试->GitHub Issue）",
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog="示例：\n"
" python demo.py 完整流程（需 OPENAI_API_KEY）\n"
" python demo.py --smoke 免 API 快速自检（仅重放+GitHub mock）\n"
" python demo.py --model gpt-5.6 临时切换模型\n"
" python demo.py --data-dir ./mine 换用自己的轨迹/架构/PRD 目录\n"
" python demo.py --create-issue 经 MCP 真实创建 Issue（需 GITHUB_TOKEN+GITHUB_REPO）\n"
"换供应商：设置 OPENAI_BASE_URL + OPENAI_MODEL 环境变量。")
parser.add_argument("--smoke", action="store_true",
help="快速自检：跳过 LLM，用内置样例仅跑重放框架+GitHub mock（无需 API Key）")
parser.add_argument("--model", default=None,
help="临时覆盖模型（等价于设置 OPENAI_MODEL；默认 gpt-5.6-luna）")
parser.add_argument("--data-dir", default=DATA, metavar="DIR",
help="输入目录：轨迹日志 trajectories.jsonl + architecture.md + PRD.md（默认 data/）")
parser.add_argument("--output", default=DEFAULT_OUTPUT, metavar="FILE",
help="GitHub Issue（mock）落盘路径（默认 output/github_issues.json）")
parser.add_argument("--create-issue", action="store_true",
help="经 MCP 在真实仓库创建 Issue（需 GITHUB_TOKEN 与 GITHUB_REPO；默认 mock 不联网）")
parser.add_argument("--no-github", action="store_true",
help="跳过步骤 4（既不 mock 也不创建 GitHub Issue）")
args = parser.parse_args()
if args.smoke:
sys.exit(run_smoke(data_dir=args.data_dir, out_path=args.output))
run_full(model=args.model, do_github=not args.no_github,
data_dir=args.data_dir, out_path=args.output,
create_issue=args.create_issue)
if __name__ == "__main__":
main()
chapter5/log-diagnosis/diagnoser.py
+159
View File
@@ -0,0 +1,159 @@
"""
diagnoser.py —— 诊断 Agent（真实调用 OpenAI）
两个阶段，均为真实 LLM 调用：
1) diagnose() 读轨迹集合 + 架构 + PRD -> 结构化问题报告(优先级/模块/描述/建议)
2) gen_test_cases() 基于问题报告 -> 生成可被 replay.py 自动执行的回归测试用例
默认模型 gpt-5.6-luna，输出走 JSON 模式，尽量稳定可解析。
"""
import json
import os
from typing import Dict, Any, List
from openai import OpenAI
MODEL = os.getenv("OPENAI_MODEL", "gpt-5.6-luna")
# --- 通用 OpenRouter 兜底 ---
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
def _map_to_openrouter_model(model: str) -> str:
"""把直连模型名映射为 OpenRouter 上的 id（非可映射 id 统一兜底到当前廉价旗舰）。"""
if not model or "/" in model:
return model or "openai/gpt-5.6-luna"
m = model.lower()
if m.startswith(("gpt-", "o1", "o3", "o4")):
return "openai/" + model
if m.startswith("claude"):
if "haiku" in m:
return "anthropic/claude-haiku-4.5"
if "sonnet" in m:
return "anthropic/claude-sonnet-4.6"
return "anthropic/claude-opus-4.8"
if m.startswith("gemini"):
return "google/" + model
return "openai/gpt-5.6-luna"
# 供 LLM 生成测试用例时使用的断言 DSL 说明（须与 replay.py 保持一致）
_ASSERTION_SPEC = """可用断言类型（assertion.type 只能取以下之一）：
- "step_present" params: {"tool": <工具名>} 该工具必须在轨迹中出现
- "tool_succeeds" params: {"tool": <工具名>} 该工具最终成功且无"多次失败后误报成功"
- "latency_under" params: {"tool": <工具名>, "threshold_ms": <整数>} 该工具单次延迟须低于阈值
- "final_status_is" params: {"value": "success"|"failed"} 任务最终状态必须等于给定值"""
class Diagnoser:
def __init__(self, model: str = MODEL):
# 通用 OpenRouter 兜底：无直连 key，或默认 gpt-5.x（直连需组织实名认证）时改走 OpenRouter。
api_key = os.getenv("OPENAI_API_KEY")
base_url = os.getenv("OPENAI_BASE_URL")
orkey = os.getenv("OPENROUTER_API_KEY")
prefer_or = bool(orkey) and (model or "").lower().startswith("gpt-5")
if prefer_or or (not api_key and orkey):
api_key, base_url, model = orkey, OPENROUTER_BASE_URL, _map_to_openrouter_model(model)
# timeout / max_retries：让偶发的网络/SSL 抖动自动重试，不至于整轮崩溃
kw = {"timeout": 60.0, "max_retries": 3}
if api_key:
kw["api_key"] = api_key
if base_url:
kw["base_url"] = base_url
self.client = OpenAI(**kw)
self.model = model
# 推理模型（gpt-5 / o 系列等）不接受 temperature=0。
self._temp = (1 if any(k in (model or "").lower()
for k in ("gpt-5", "o1", "o3", "o4", "thinking", "reasoner", "kimi-k3"))
else 0)
# ---------- 阶段一：诊断 ----------
def diagnose(self, architecture: str, prd: str,
trajectories: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
traj_text = json.dumps(trajectories, ensure_ascii=False, indent=2)
system = (
"你是资深的 Agent 系统诊断专家。给定系统架构文档、PRD 与一组生产轨迹，"
"你要判断每条轨迹的执行流程是否符合架构与 PRD 的要求，识别问题模式、定位到具体模块，"
"输出结构化问题报告。只报告确有证据的问题，不要臆造。"
)
user = f"""# 系统架构
{architecture}
# PRD
{prd}
# 生产轨迹集合（JSON）
{traj_text}
# 任务
逐条核对轨迹与 PRD/架构，找出偏离项。以 JSON 输出，结构：
{{
"problems": [
{{
"title": "一句话问题标题",
"priority": "P0|P1|P2|P3",
"module": "涉及的模块名(取架构中的模块)",
"description": "问题描述，引用具体轨迹与轮次作为证据",
"suggestion": "可操作的改进建议",
"trajectory_ids": ["涉及的轨迹ID"],
"focus_turns": [关键交互轮次的index],
"prd_ref": "对应的PRD条目(如 R1/R2/R3)",
"suggested_assignee": "建议负责人(可留空)"
}}
]
}}
只输出 JSON。"""
resp = self.client.chat.completions.create(
model=self.model,
messages=[{"role": "system", "content": system},
{"role": "user", "content": user}],
response_format={"type": "json_object"},
temperature=self._temp,
)
try:
data = json.loads(resp.choices[0].message.content)
except json.JSONDecodeError:
data = {}
return data.get("problems", [])
# ---------- 阶段二：生成回归测试用例 ----------
def gen_test_cases(self, problems: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
prob_text = json.dumps(problems, ensure_ascii=False, indent=2)
system = (
"你是测试工程师。基于诊断出的问题，为每个问题生成一条回归测试用例，"
"用例引用问题轨迹 ID 与关键交互轮次，并给出一个可被自动重放框架求值的断言。"
)
user = f"""# 已诊断的问题
{prob_text}
# 断言 DSL
{_ASSERTION_SPEC}
# 任务
为每个问题生成 1 条回归测试用例。断言应表达"修复后系统应满足的正确行为"
（例如：退款前应出现 verify_refund_eligibility；process_refund 应最终成功；check_stock 延迟应 < 5000ms）。
以 JSON 输出：
{{
"test_cases": [
{{
"test_id": "RT-001",
"trajectory_id": "引用的问题轨迹ID",
"focus_turn": 关键轮次index,
"description": "该用例验证什么",
"assertion": {{"type": "...", "params": {{...}}}}
}}
]
}}
只输出 JSON。"""
resp = self.client.chat.completions.create(
model=self.model,
messages=[{"role": "system", "content": system},
{"role": "user", "content": user}],
response_format={"type": "json_object"},
temperature=self._temp,
)
try:
data = json.loads(resp.choices[0].message.content)
except json.JSONDecodeError:
data = {}
return data.get("test_cases", [])
chapter5/log-diagnosis/env.example
+21
View File
@@ -0,0 +1,21 @@
# 复制为 .env 后填入你的 OpenAI Key（直连，必填其一）
OPENAI_API_KEY=your-openai-api-key
# 通用兜底：未配置 OPENAI_API_KEY 时自动改走 OpenRouter；
# 默认模型 gpt-5.6-luna（gpt-5.x）直连 OpenAI 需组织实名认证，
# 故设置了本 key 时会优先走 OpenRouter（route openai/gpt-5.6-luna）。
# OPENROUTER_API_KEY=your-openrouter-api-key
# 可选：更换模型（默认 gpt-5.6-luna）
# OPENAI_MODEL=gpt-5.6-luna
# 可选：换供应商（任何兼容 OpenAI 接口的服务，如 Moonshot / 火山方舟 / 本地 vLLM）
# 同时把 OPENAI_API_KEY 换成该供应商的 Key、OPENAI_MODEL 换成其模型名
# OPENAI_BASE_URL=https://api.moonshot.cn/v1
# 可选：真实接入 GitHub MCP 时使用（默认 mock，无需填写）
# 需配合 python demo.py --create-issue 才会真正联网创建 Issue
# GITHUB_TOKEN=your-github-token
# GITHUB_REPO=owner/repo
# 可选：覆盖 GitHub MCP Server 启动命令（默认用官方 Docker 镜像）
# GITHUB_MCP_COMMAND=docker run -i --rm -e GITHUB_PERSONAL_ACCESS_TOKEN ghcr.io/github/github-mcp-server
chapter5/log-diagnosis/github_mcp.py
+141
View File
@@ -0,0 +1,141 @@
"""
github_mcp.py —— GitHub Issue 创建（默认 mock，可选真实 MCP）
- mock（默认）：把"创建 Issue"渲染成将要提交的 Issue 结构，打印并写入本地文件，
不联网、不需要 token。
- 真实（mock=False，需 GITHUB_TOKEN + GITHUB_REPO）：通过 MCP 协议连接官方
GitHub MCP Server（stdio），调用其 `create_issue` 工具在真实仓库创建 Issue。
出于安全，真实创建须由 demo.py 的 --create-issue 显式开启。
"""
import json
import os
import shlex
from datetime import datetime
from typing import Dict, Any, List
_OUT = os.path.join(os.path.dirname(__file__), "output", "github_issues.json")
# 官方 GitHub MCP Server 的默认启动命令（可用 GITHUB_MCP_COMMAND 覆盖）。
# 默认用 Docker 运行官方镜像；也可换成任何暴露 create_issue 工具的 MCP Server。
_DEFAULT_MCP_COMMAND = (
"docker run -i --rm -e GITHUB_PERSONAL_ACCESS_TOKEN ghcr.io/github/github-mcp-server")
# 优先级 -> GitHub label 的映射
_PRIORITY_LABEL = {"P0": "priority:critical", "P1": "priority:high",
"P2": "priority:medium", "P3": "priority:low"}
def build_issue(problem: Dict[str, Any], test_cases: List[Dict[str, Any]]) -> Dict[str, Any]:
"""把一条诊断问题 + 关联回归测试用例，渲染成 GitHub Issue 结构。"""
prio = problem.get("priority", "P2")
module = problem.get("module", "unknown")
related = [tc for tc in test_cases
if tc.get("trajectory_id") in problem.get("trajectory_ids", [])]
body_lines = [
f"## 问题描述\n{problem.get('description', '')}",
f"\n## 涉及模块\n`{module}`",
f"\n## 优先级\n{prio}",
f"\n## 改进建议\n{problem.get('suggestion', '')}",
f"\n## 相关生产轨迹\n" + ", ".join(problem.get("trajectory_ids", []) or ["(无)"]),
]
if related:
body_lines.append("\n## 关联回归测试用例")
for tc in related:
body_lines.append(
f"- `{tc.get('test_id')}` (轨迹 {tc.get('trajectory_id')} "
f"第 {tc.get('focus_turn')} 轮): {tc.get('description', '')}")
return {
"title": f"[{prio}][{module}] {problem.get('title', problem.get('description', ''))[:60]}",
"body": "\n".join(body_lines),
"labels": [f"module:{module}", _PRIORITY_LABEL.get(prio, "priority:medium"),
"auto-diagnosis"],
"assignees": [problem.get("suggested_assignee", "")] if problem.get("suggested_assignee") else [],
}
def create_issues(problems: List[Dict[str, Any]], test_cases: List[Dict[str, Any]],
mock: bool = True, out_path: str = _OUT,
repo: str = None, token: str = None) -> List[Dict[str, Any]]:
"""为每条问题创建 Issue。
mock=True（默认）：打印 + 落盘到 out_path，不联网。
mock=False：通过 GitHub MCP Server 在 repo（owner/repo）真实创建 Issue，需 token。
"""
issues = [build_issue(p, test_cases) for p in problems]
if mock:
os.makedirs(os.path.dirname(out_path) or ".", exist_ok=True)
payload = {"created_at": datetime.now().isoformat(),
"mode": "mock", "issues": issues}
with open(out_path, "w", encoding="utf-8") as f:
json.dump(payload, f, ensure_ascii=False, indent=2)
print(f"\n[github_mcp:mock] 已将 {len(issues)} 个 Issue 写入 {out_path}")
for i, iss in enumerate(issues, 1):
print(f"\n----- Mock GitHub Issue #{i} -----")
print(f"title : {iss['title']}")
print(f"labels : {iss['labels']}")
print("body :")
for ln in iss["body"].splitlines():
print(" " + ln)
else:
if not token or not repo:
raise RuntimeError(
"真实创建需 GITHUB_TOKEN 与 GITHUB_REPO(owner/repo)，见 README。")
created = _create_issues_via_mcp(issues, repo=repo, token=token)
print(f"\n[github_mcp] 通过 MCP 在 {repo} 创建了 {len(created)} 个 Issue：")
for url in created:
print(f" {url}")
return issues
def _create_issues_via_mcp(issues: List[Dict[str, Any]], repo: str, token: str) -> List[str]:
"""通过 stdio 连接官方 GitHub MCP Server，逐个调用 create_issue 工具。
返回创建成功的 Issue URL 列表。需要已安装 `mcp` Python SDK 与可用的
GitHub MCP Server（默认 Docker 镜像，可用 GITHUB_MCP_COMMAND 覆盖启动命令）。
"""
import asyncio
try:
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
except ImportError as e: # pragma: no cover - 依赖缺失时给出清晰指引
raise RuntimeError(
"缺少 MCP 客户端：pip install mcp（并确保 GitHub MCP Server 可启动）") from e
owner, _, name = repo.partition("/")
if not owner or not name:
raise RuntimeError(f"GITHUB_REPO 需形如 owner/repo，收到：{repo!r}")
cmd = shlex.split(os.getenv("GITHUB_MCP_COMMAND", _DEFAULT_MCP_COMMAND))
params = StdioServerParameters(
command=cmd[0], args=cmd[1:],
env={**os.environ, "GITHUB_PERSONAL_ACCESS_TOKEN": token})
async def _run() -> List[str]:
urls: List[str] = []
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
for iss in issues:
result = await session.call_tool("create_issue", {
"owner": owner, "repo": name,
"title": iss["title"], "body": iss["body"],
"labels": iss["labels"],
"assignees": iss["assignees"],
})
# MCP 工具返回文本内容；尽力提取 Issue URL，否则回退为原始文本。
text = "".join(getattr(c, "text", "") for c in result.content)
url = text
try:
url = json.loads(text).get("html_url", text)
except Exception:
pass
urls.append(url)
return urls
return asyncio.run(_run())
chapter5/log-diagnosis/http_service.py
+98
View File
@@ -0,0 +1,98 @@
"""Small real HTTP system used by the Experiment 5-8 campaign.
The service intentionally exposes ordinary order, refund, and inventory
endpoints. The buggy/fixed distinction lives in the orchestrator under test:
the buggy orchestrator skips a required endpoint and mishandles an inventory
timeout, while the fixed orchestrator makes the required calls and degrades.
"""
from __future__ import annotations
import argparse
import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse
class Handler(BaseHTTPRequestHandler):
server_version = "Experiment58HTTP/1.0"
def log_message(self, fmt: str, *args: object) -> None:
print(
json.dumps(
{
"client": self.client_address[0],
"message": fmt % args,
"time": time.time(),
},
ensure_ascii=False,
),
flush=True,
)
def _json(self, status: int, value: object) -> None:
raw = json.dumps(value, ensure_ascii=False).encode("utf-8")
self.send_response(status)
self.send_header("content-type", "application/json; charset=utf-8")
self.send_header("content-length", str(len(raw)))
self.end_headers()
try:
self.wfile.write(raw)
except BrokenPipeError:
# A timed-out client is an expected part of the observed buggy run.
pass
def _body(self) -> dict[str, object]:
length = int(self.headers.get("content-length") or 0)
if not length:
return {}
value = json.loads(self.rfile.read(length))
return value if isinstance(value, dict) else {}
def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
parsed = urlparse(self.path)
if parsed.path == "/health":
self._json(200, {"ok": True})
return
if parsed.path.startswith("/orders/"):
order_id = parsed.path.rsplit("/", 1)[-1]
self._json(200, {"order_id": order_id, "status": "paid", "sku": "SKU-42"})
return
if parsed.path.startswith("/inventory/"):
sku = parsed.path.rsplit("/", 1)[-1]
degraded = parse_qs(parsed.query).get("degraded", ["0"])[0] == "1"
if degraded:
self._json(200, {"sku": sku, "stock": 12, "source": "cache", "degraded": True})
else:
# Longer than the campaign's real client deadline.
time.sleep(0.8)
self._json(200, {"sku": sku, "stock": 12, "source": "origin", "degraded": False})
return
self._json(404, {"error": "not_found"})
def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
body = self._body()
if self.path == "/refund/eligibility":
self._json(200, {"order_id": body.get("order_id"), "eligible": True})
return
if self.path == "/refund/process":
self._json(200, {"order_id": body.get("order_id"), "refund_id": "RF-LIVE-1"})
return
if self.path == "/notifications":
self._json(200, {"sent": True, "status": body.get("status")})
return
self._json(404, {"error": "not_found"})
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--port", type=int, required=True)
args = parser.parse_args()
server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
print(json.dumps({"listening": args.port}), flush=True)
server.serve_forever()
if __name__ == "__main__":
main()
chapter5/log-diagnosis/replay.py
+138
View File
@@ -0,0 +1,138 @@
"""
replay.py —— 回归测试重放框架
输入：Agent 生成的回归测试用例（结构化 JSON），引用轨迹 ID 与交互轮次。
过程：从原始轨迹取出该任务的输入，喂给被测系统（sut.run_task）重放，
在重放产生的新轨迹上求值断言，给出 通过/失败。
一个测试用例的结构（Agent 需按此 DSL 生成）：
{
"test_id": "RT-001",
"trajectory_id": "T-1001", # 引用的问题轨迹
"focus_turn": 3, # 关键交互轮次（问题所在）
"description": "退款前必须先做资格校验",
"assertion": {"type": "step_present", "params": {"tool": "verify_refund_eligibility"}}
}
支持的断言类型（replay 框架内置，可被自动求值）：
- step_present params.tool 某工具在轨迹中必须出现（如强制前置校验）
- tool_succeeds params.tool 某工具最终必须成功、且不得出现连续失败后误报成功
- latency_under params.tool, threshold_ms 某工具单次延迟必须低于阈值
- final_status_is params.value 任务最终状态必须等于给定值
"""
import json
import os
from typing import Dict, Any, List, Tuple
import sut
_DATA = os.path.join(os.path.dirname(__file__), "data", "trajectories.jsonl")
def load_trajectories(path: str = _DATA) -> Dict[str, Dict[str, Any]]:
"""读取生产轨迹集合，按 trajectory_id 索引。"""
out = {}
with open(path, "r", encoding="utf-8") as f:
for line in f:
line = line.strip()
if not line:
continue
t = json.loads(line)
out[t["trajectory_id"]] = t
return out
# ---------------- 断言求值器 ----------------
def _tool_turns(traj: Dict[str, Any], tool: str) -> List[Dict[str, Any]]:
return [t for t in traj.get("turns", []) if t.get("tool") == tool]
def _eval_assertion(assertion: Dict[str, Any], traj: Dict[str, Any]) -> Tuple[bool, str]:
a_type = assertion.get("type")
params = assertion.get("params", {})
if a_type == "step_present":
tool = params.get("tool") or params.get("step")
ok = len(_tool_turns(traj, tool)) > 0
return ok, f"工具 {tool} {'出现' if ok else '缺失'}"
if a_type == "tool_succeeds":
tool = params.get("tool")
calls = _tool_turns(traj, tool)
if not calls:
return False, f"{tool} 未被调用"
n_err = sum(1 for c in calls if c.get("status") == "error")
last_ok = calls[-1].get("status") == "success"
# 修复标准：最终成功，且不存在"多次失败后仍误报成功"（>=2 次失败视为未正确处理）
ok = last_ok and n_err < 2
return ok, f"{tool} 调用 {len(calls)} 次, 失败 {n_err} 次, 末次{'成功' if last_ok else '失败'}"
if a_type == "latency_under":
tool = params.get("tool")
thr = params.get("threshold_ms") or params.get("threshold")
if thr is None:
return False, "latency_under 断言缺失阈值设置"
try:
thr = float(thr)
except (TypeError, ValueError):
return False, f"latency_under 阈值非法: {thr!r}"
calls = _tool_turns(traj, tool)
if not calls:
return False, f"{tool} 未被调用"
latencies = []
for c in calls:
lat = c.get("latency_ms")
if lat is None:
lat = 0
try:
latencies.append(float(lat))
except (TypeError, ValueError):
latencies.append(0.0)
worst = max(latencies) if latencies else 0.0
ok = worst < thr
return ok, f"{tool} 最大延迟 {worst}ms, 阈值 {thr}ms"
if a_type == "final_status_is":
want = params.get("value")
ok = traj.get("final_status") == want
return ok, f"final_status={traj.get('final_status')}, 期望={want}"
return False, f"未知断言类型: {a_type}"
def run_test_case(tc: Dict[str, Any], trajectories: Dict[str, Any],
fixed: bool) -> Dict[str, Any]:
"""对单条测试用例：取原始轨迹输入 -> 重放被测系统 -> 求值断言。"""
tid = tc.get("trajectory_id")
src = trajectories.get(tid)
if src is None:
return {"test_id": tc.get("test_id"), "passed": False,
"detail": f"引用的轨迹 {tid} 不存在"}
replayed = sut.run_task(src["task_input"], fixed=fixed)
passed, detail = _eval_assertion(tc.get("assertion", {}), replayed)
return {
"test_id": tc.get("test_id"),
"trajectory_id": tid,
"focus_turn": tc.get("focus_turn"),
"passed": passed,
"detail": detail,
"replay_mode": "fixed" if fixed else "buggy",
}
def run_suite(test_cases: List[Dict[str, Any]], fixed: bool,
path: str = _DATA) -> List[Dict[str, Any]]:
"""跑完整套测试用例，返回结果列表。"""
trajectories = load_trajectories(path)
results = []
for tc in test_cases:
try:
results.append(run_test_case(tc, trajectories, fixed))
except Exception as e: # 单条用例出错不影响整套
results.append({"test_id": tc.get("test_id"), "passed": False,
"detail": f"用例执行异常: {e}",
"replay_mode": "fixed" if fixed else "buggy"})
return results
chapter5/log-diagnosis/requirements.txt
+4
View File
@@ -0,0 +1,4 @@
openai>=1.30.0
python-dotenv>=1.0.0
# 仅 --create-issue（真实经 MCP 创建 GitHub Issue）时需要；mock/自检不需要
mcp>=1.0.0
chapter5/log-diagnosis/sut.py
+111
View File
@@ -0,0 +1,111 @@
"""
sut.py —— System Under Test（被测系统的确定性仿真）
回归测试的核心是"用相同输入重放，断言修复后系统能产生正确行为"。
这里用一个**确定性**的仿真器来扮演线上 Agent 系统：
- run_task(task_input, fixed=False)：复现线上（有 bug）的行为，
产出的轨迹会带上和生产轨迹一致的三类已知问题。
- run_task(task_input, fixed=True)：模拟"修复后"的系统，
正确执行前置校验 / 重试退避 / 库存降级。
replay.py 会分别对 fixed=False / fixed=True 重放同一输入，
从而演示同一条回归测试用例的"失败(复现bug)"与"通过(验证修复)"。
轨迹结构与 data/trajectories.jsonl 完全一致，便于对比。
"""
from typing import Dict, Any
def run_task(task_input: Dict[str, Any], fixed: bool = False) -> Dict[str, Any]:
"""给定任务输入，确定性地跑一遍被测系统，返回一条轨迹。"""
intent = task_input.get("intent")
order_id = task_input.get("order_id", "UNKNOWN")
turns = []
idx = 0
def add(**kw):
nonlocal idx
kw["index"] = idx
idx += 1
turns.append(kw)
# 0. 用户输入 & 意图识别
add(role="user", content=f"task={intent}, order={order_id}")
add(role="assistant", module="intent_parser", content=f"意图={intent}")
final_status = "success"
if intent == "refund":
# 查询订单
add(role="tool", module="order_service", tool="query_order",
input={"order_id": order_id},
output={"status": task_input.get("order_status", "paid")},
status="success", latency_ms=210)
# R1：退款前置资格校验（仅修复版本执行）
if fixed:
add(role="tool", module="order_service", tool="verify_refund_eligibility",
input={"order_id": order_id},
output={"eligible": True}, status="success", latency_ms=120)
# R2：支付重试 + 退避
if task_input.get("payment_flaky") and not fixed:
# 线上 bug：无退避，连续失败后误报成功
for _ in range(3):
add(role="tool", module="payment_service", tool="process_refund",
input={"order_id": order_id},
output={"error": "gateway_timeout"},
status="error", latency_ms=3000)
add(role="assistant", module="payment_service",
content="多次失败，仍按成功结束（bug）")
final_status = "success" # 误报成功
elif task_input.get("payment_flaky") and fixed:
# 修复：一次失败后带退避重试成功
add(role="tool", module="payment_service", tool="process_refund",
input={"order_id": order_id},
output={"error": "gateway_timeout"}, status="error", latency_ms=1500)
add(role="assistant", module="payment_service", content="退避 800ms 后重试")
add(role="tool", module="payment_service", tool="process_refund",
input={"order_id": order_id, "retry": 1},
output={"refund_id": "R-OK"}, status="success", latency_ms=600)
else:
add(role="tool", module="payment_service", tool="process_refund",
input={"order_id": order_id},
output={"refund_id": "R-OK"}, status="success", latency_ms=540)
elif intent == "order_status":
add(role="tool", module="order_service", tool="query_order",
input={"order_id": order_id},
output={"status": "paid", "sku": task_input.get("sku")},
status="success", latency_ms=220)
# R3：库存查询延迟
if task_input.get("slow_inventory") and not fixed:
# 线上 bug：超时仍阻塞等待，不降级
add(role="tool", module="inventory_service", tool="check_stock",
input={"sku": task_input.get("sku")},
output={"stock": 12}, status="success", latency_ms=8300)
elif task_input.get("slow_inventory") and fixed:
# 修复：超过阈值走降级路径，快速返回
add(role="tool", module="inventory_service", tool="check_stock",
input={"sku": task_input.get("sku"), "degraded": True},
output={"stock": "cached:12", "degraded": True},
status="success", latency_ms=400)
else:
add(role="tool", module="inventory_service", tool="check_stock",
input={"sku": task_input.get("sku")},
output={"stock": 5}, status="success", latency_ms=300)
# R4：通知用户
add(role="tool", module="notification_service", tool="notify_user",
input={"final_status": final_status},
output={"sent": True}, status="success", latency_ms=60)
return {
"trajectory_id": f"REPLAY::{order_id}::{'fixed' if fixed else 'buggy'}",
"task_input": task_input,
"final_status": final_status,
"turns": turns,
}
chapter5/log-diagnosis/test_campaign_orchestrator.py
+64
View File
@@ -0,0 +1,64 @@
"""Live HTTP regressions for the findings reported in issue #502."""
from __future__ import annotations
import threading
from collections.abc import Iterator
from http.server import ThreadingHTTPServer
import pytest
from campaign import INVENTORY_DEADLINE_SECONDS, _trajectory
from http_service import Handler
@pytest.fixture
def order_service() -> Iterator[str]:
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
server.daemon_threads = True
thread = threading.Thread(target=server.serve_forever, daemon=True)
thread.start()
host, port = server.server_address
try:
yield f"http://{host}:{port}"
finally:
server.shutdown()
server.server_close()
thread.join(timeout=1)
def test_refund_verifies_eligibility_before_processing(order_service: str) -> None:
trajectory = _trajectory(
order_service,
{"intent": "refund", "order_id": "ORD-58-A"},
"HTTP-RF-001",
)
tools = [turn.get("tool") for turn in trajectory["turns"]]
assert tools.count("verify_refund_eligibility") == 1
assert tools.index("verify_refund_eligibility") < tools.index("process_refund")
assert trajectory["final_status"] == "success"
def test_inventory_timeout_uses_cache_within_deadline(order_service: str) -> None:
trajectory = _trajectory(
order_service,
{
"intent": "order_status",
"order_id": "ORD-58-B",
"sku": "SKU-42",
},
"HTTP-INV-001",
)
stock_calls = [
turn for turn in trajectory["turns"] if turn.get("tool") == "check_stock"
]
assert len(stock_calls) == 2
assert stock_calls[0]["status"] == "error"
assert stock_calls[1]["status"] == "success"
assert stock_calls[1]["response"]["source"] == "cache"
assert sum(float(turn["latency_ms"]) for turn in stock_calls) < (
INVENTORY_DEADLINE_SECONDS * 1000
)
assert trajectory["final_status"] == "success"
chapter5/log-diagnosis/test_github_mcp_bare_output.py
+24
View File
@@ -0,0 +1,24 @@
import json
import os
import github_mcp
def test_create_issues_bare_filename(tmp_path, monkeypatch):
"""out_path 为不带目录的裸文件名时，落盘不应崩溃（dirname 为空串）。"""
monkeypatch.chdir(tmp_path)
problems = [{"priority": "P1", "module": "refund",
"title": "退款未校验", "description": "描述"}]
issues = github_mcp.create_issues(problems, [], mock=True,
out_path="issues.json")
assert os.path.exists("issues.json")
with open("issues.json", encoding="utf-8") as f:
payload = json.load(f)
assert payload["issues"] == issues
def test_create_issues_with_directory(tmp_path):
"""带目录的 out_path 保持原行为：自动创建父目录。"""
out = tmp_path / "sub" / "issues.json"
github_mcp.create_issues([], [], mock=True, out_path=str(out))
assert out.exists()
chapter5/log-diagnosis/test_replay_latency.py
+25
View File
@@ -0,0 +1,25 @@
from replay import _eval_assertion
def test_latency_under_null_latency_ms():
assertion = {
"type": "latency_under",
"params": {"tool": "test_tool", "threshold": 150},
}
traj = {
"turns": [
{"tool": "test_tool", "latency_ms": None},
{"tool": "test_tool", "latency_ms": 120},
]
}
ok, msg = _eval_assertion(assertion, traj)
assert ok
assert "最大延迟 120.0ms" in msg
def test_latency_under_missing_threshold():
assertion = {"type": "latency_under", "params": {"tool": "test_tool"}}
traj = {"turns": [{"tool": "test_tool", "latency_ms": 100}]}
ok, msg = _eval_assertion(assertion, traj)
assert not ok
assert "断言缺失阈值设置" in msg
chapter5/log-diagnosis/validation/latest.json
+7
View File
@@ -0,0 +1,7 @@
{
"experiment": "5-8",
"run_id": "exp5-8-live-http-mcp-20260730-053403",
"manifest": "validation/runs/exp5-8-live-http-mcp-20260730-053403/manifest.json",
"manifest_sha256": "68e09e7c8b4fc100e0612a6f81978079c393a822025bfa794b6dda85134a5813",
"official_complete": true
}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053056/PRD.md
+8
View File
@@ -0,0 +1,8 @@
# Live diagnosis experiment PRD
- R1 (P0): Every refund must call `verify_refund_eligibility` before
`process_refund`; a refund without the check is a policy violation.
- R2 (P1): `check_stock` must complete within 250 ms. On origin timeout the
orchestrator must use the degraded cache route; it must not simply fail.
- R3 (P1): Regression cases must cite the source trajectory ID and the exact
observed turn where the violation is visible.
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053056/architecture.md
+7
View File
@@ -0,0 +1,7 @@
# Live diagnosis experiment architecture
The orchestrator calls a local HTTP order service. Refund flows MUST call
`verify_refund_eligibility` before `process_refund`. Inventory origin calls
have a 250 ms client deadline; on timeout the orchestrator MUST call the same
`check_stock` operation through the degraded cache route and finish normally.
Every trajectory turn records its measured HTTP latency and raw response.
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053056/http_service.py
+98
View File
@@ -0,0 +1,98 @@
"""Small real HTTP system used by the Experiment 5-8 campaign.
The service intentionally exposes ordinary order, refund, and inventory
endpoints. The buggy/fixed distinction lives in the orchestrator under test:
the buggy orchestrator skips a required endpoint and mishandles an inventory
timeout, while the fixed orchestrator makes the required calls and degrades.
"""
from __future__ import annotations
import argparse
import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse
class Handler(BaseHTTPRequestHandler):
server_version = "Experiment58HTTP/1.0"
def log_message(self, fmt: str, *args: object) -> None:
print(
json.dumps(
{
"client": self.client_address[0],
"message": fmt % args,
"time": time.time(),
},
ensure_ascii=False,
),
flush=True,
)
def _json(self, status: int, value: object) -> None:
raw = json.dumps(value, ensure_ascii=False).encode("utf-8")
self.send_response(status)
self.send_header("content-type", "application/json; charset=utf-8")
self.send_header("content-length", str(len(raw)))
self.end_headers()
try:
self.wfile.write(raw)
except BrokenPipeError:
# A timed-out client is an expected part of the observed buggy run.
pass
def _body(self) -> dict[str, object]:
length = int(self.headers.get("content-length") or 0)
if not length:
return {}
value = json.loads(self.rfile.read(length))
return value if isinstance(value, dict) else {}
def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
parsed = urlparse(self.path)
if parsed.path == "/health":
self._json(200, {"ok": True})
return
if parsed.path.startswith("/orders/"):
order_id = parsed.path.rsplit("/", 1)[-1]
self._json(200, {"order_id": order_id, "status": "paid", "sku": "SKU-42"})
return
if parsed.path.startswith("/inventory/"):
sku = parsed.path.rsplit("/", 1)[-1]
degraded = parse_qs(parsed.query).get("degraded", ["0"])[0] == "1"
if degraded:
self._json(200, {"sku": sku, "stock": 12, "source": "cache", "degraded": True})
else:
# Longer than the campaign's real client deadline.
time.sleep(0.8)
self._json(200, {"sku": sku, "stock": 12, "source": "origin", "degraded": False})
return
self._json(404, {"error": "not_found"})
def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
body = self._body()
if self.path == "/refund/eligibility":
self._json(200, {"order_id": body.get("order_id"), "eligible": True})
return
if self.path == "/refund/process":
self._json(200, {"order_id": body.get("order_id"), "refund_id": "RF-LIVE-1"})
return
if self.path == "/notifications":
self._json(200, {"sent": True, "status": body.get("status")})
return
self._json(404, {"error": "not_found"})
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--port", type=int, required=True)
args = parser.parse_args()
server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
print(json.dumps({"listening": args.port}), flush=True)
server.serve_forever()
if __name__ == "__main__":
main()
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053056/production_trajectories.jsonl
+2
View File
@@ -0,0 +1,2 @@
{"trajectory_id": "HTTP-RF-001::buggy", "source_trajectory_id": "HTTP-RF-001", "implementation": "buggy", "task_input": {"intent": "refund", "order_id": "ORD-58-A"}, "final_status": "success", "turns": [{"index": 0, "role": "user", "content": "{\"intent\": \"refund\", \"order_id\": \"ORD-58-A\"}"}, {"index": 1, "role": "tool", "module": "http_order_system", "tool": "query_order", "method": "GET", "path": "/orders/ORD-58-A", "request": null, "http_status": 200, "status": "success", "latency_ms": 0.451, "response": {"order_id": "ORD-58-A", "status": "paid", "sku": "SKU-42"}}, {"index": 2, "role": "tool", "module": "http_order_system", "tool": "process_refund", "method": "POST", "path": "/refund/process", "request": {"order_id": "ORD-58-A"}, "http_status": 200, "status": "success", "latency_ms": 0.431, "response": {"order_id": "ORD-58-A", "refund_id": "RF-LIVE-1"}}, {"index": 3, "role": "tool", "module": "http_order_system", "tool": "notify_user", "method": "POST", "path": "/notifications", "request": {"status": "success"}, "http_status": 200, "status": "success", "latency_ms": 0.423, "response": {"sent": true, "status": "success"}}]}
{"trajectory_id": "HTTP-INV-001::buggy", "source_trajectory_id": "HTTP-INV-001", "implementation": "buggy", "task_input": {"intent": "order_status", "order_id": "ORD-58-B", "sku": "SKU-42"}, "final_status": "failed", "turns": [{"index": 0, "role": "user", "content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"}, {"index": 1, "role": "tool", "module": "http_order_system", "tool": "query_order", "method": "GET", "path": "/orders/ORD-58-B", "request": null, "http_status": 200, "status": "success", "latency_ms": 0.378, "response": {"order_id": "ORD-58-B", "status": "paid", "sku": "SKU-42"}}, {"index": 2, "role": "tool", "module": "http_order_system", "tool": "check_stock", "method": "GET", "path": "/inventory/SKU-42", "request": null, "http_status": null, "status": "error", "latency_ms": 352.364, "error": "TimeoutError: timed out"}, {"index": 3, "role": "tool", "module": "http_order_system", "tool": "notify_user", "method": "POST", "path": "/notifications", "request": {"status": "failed"}, "http_status": 200, "status": "success", "latency_ms": 2.174, "response": {"sent": true, "status": "failed"}}]}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053056/service.stdout.jsonl
+8
View File
@@ -0,0 +1,8 @@
{"listening": 56305}
{"client": "127.0.0.1", "message": "\"GET /health HTTP/1.1\" 200 -", "time": 1785360656.4696589}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-A HTTP/1.1\" 200 -", "time": 1785360656.470422}
{"client": "127.0.0.1", "message": "\"POST /refund/process HTTP/1.1\" 200 -", "time": 1785360656.470944}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785360656.471394}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785360656.4718232}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785360656.8262029}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785360657.274664}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/PRD.md
+8
View File
@@ -0,0 +1,8 @@
# Live diagnosis experiment PRD
- R1 (P0): Every refund must call `verify_refund_eligibility` before
`process_refund`; a refund without the check is a policy violation.
- R2 (P1): `check_stock` must complete within 250 ms. On origin timeout the
orchestrator must use the degraded cache route; it must not simply fail.
- R3 (P1): Regression cases must cite the source trajectory ID and the exact
observed turn where the violation is visible.
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/architecture.md
+7
View File
@@ -0,0 +1,7 @@
# Live diagnosis experiment architecture
The orchestrator calls a local HTTP order service. Refund flows MUST call
`verify_refund_eligibility` before `process_refund`. Inventory origin calls
have a 250 ms client deadline; on timeout the orchestrator MUST call the same
`check_stock` operation through the degraded cache route and finish normally.
Every trajectory turn records its measured HTTP latency and raw response.
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/diagnosis.json
+34
View File
@@ -0,0 +1,34 @@
{
"problems": [
{
"title": "Refund flow missing verify_refund_eligibility check",
"priority": "P0",
"module": "http_order_system",
"description": "Refund flow did not call `verify_refund_eligibility` before `process_refund`, violating the policy requirement that every refund must include this check.",
"suggestion": "Add a call to the `verify_refund_eligibility` tool prior to the `process_refund` step in the refund workflow.",
"prd_ref": "R1",
"trajectory_ids": [
"HTTP-RF-001"
],
"focus_turns": [
2
],
"suggested_assignee": "order-service-team"
},
{
"title": "check_stock origin timeout not handled via degraded cache route",
"priority": "P1",
"module": "http_order_system",
"description": "check_stock call timed out (latency 350.414 ms exceeds 250 ms client deadline) but orchestrator failed to use the degraded cache route, resulting in flow failure.",
"suggestion": "Implement logic to invoke `check_stock` through the degraded cache route when the origin call times out, ensuring the flow completes normally.",
"prd_ref": "R2",
"trajectory_ids": [
"HTTP-INV-001"
],
"focus_turns": [
2
],
"suggested_assignee": "inventory-service-team"
}
]
}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/github_mcp_receipt.json
+18
View File
@@ -0,0 +1,18 @@
{
"server": "official github/github-mcp-server",
"transport": "stdio",
"tool": "issue_write(method=create)",
"request": {
"method": "create",
"owner": "bojieli",
"repo": "ai-agent-book",
"title": "[Experiment 5-8][auto-diagnosis] Live HTTP regression findings (exp5-8-live-http-mcp-20260730-053403)",
"body": "This issue was created automatically by the Chapter 5 Experiment 5-8 acceptance campaign.\n\n## Evidence-backed diagnosis\n\n- **P0 Refund flow missing verify_refund_eligibility check** (R1): Refund flow did not call `verify_refund_eligibility` before `process_refund`, violating the policy requirement that every refund must include this check. Trajectories: HTTP-RF-001; turns: [2].\n- **P1 check_stock origin timeout not handled via degraded cache route** (R2): check_stock call timed out (latency 350.414 ms exceeds 250 ms client deadline) but orchestrator failed to use the degraded cache route, resulting in flow failure. Trajectories: HTTP-INV-001; turns: [2].\n\n## Generated regression tests\n\n- `TC-R1-HTTP-RF-001-T2` cites `HTTP-RF-001` turn 2: `{\"type\": \"step_present\", \"params\": {\"tool\": \"verify_refund_eligibility\"}}`\n- `TC-R2-LATENCY-HTTP-INV-001-T2` cites `HTTP-INV-001` turn 2: `{\"type\": \"latency_under\", \"params\": {\"tool\": \"check_stock\", \"threshold_ms\": 250}}`\n- `TC-R2-STATUS-HTTP-INV-001-T2` cites `HTTP-INV-001` turn 2: `{\"type\": \"final_status_is\", \"params\": {\"value\": \"success\"}}`\n\nThe campaign replayed each test against the live buggy and fixed HTTP orchestrators: all tests failed on buggy behavior and passed on fixed behavior."
},
"response": {
"is_error": false,
"content": "{\"id\":\"5013792084\",\"url\":\"https://github.com/bojieli/ai-agent-book/issues/502\"}"
},
"issue_url": "https://github.com/bojieli/ai-agent-book/issues/502",
"credential_free": true
}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/http_service.py
+98
View File
@@ -0,0 +1,98 @@
"""Small real HTTP system used by the Experiment 5-8 campaign.
The service intentionally exposes ordinary order, refund, and inventory
endpoints. The buggy/fixed distinction lives in the orchestrator under test:
the buggy orchestrator skips a required endpoint and mishandles an inventory
timeout, while the fixed orchestrator makes the required calls and degrades.
"""
from __future__ import annotations
import argparse
import json
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
from urllib.parse import parse_qs, urlparse
class Handler(BaseHTTPRequestHandler):
server_version = "Experiment58HTTP/1.0"
def log_message(self, fmt: str, *args: object) -> None:
print(
json.dumps(
{
"client": self.client_address[0],
"message": fmt % args,
"time": time.time(),
},
ensure_ascii=False,
),
flush=True,
)
def _json(self, status: int, value: object) -> None:
raw = json.dumps(value, ensure_ascii=False).encode("utf-8")
self.send_response(status)
self.send_header("content-type", "application/json; charset=utf-8")
self.send_header("content-length", str(len(raw)))
self.end_headers()
try:
self.wfile.write(raw)
except BrokenPipeError:
# A timed-out client is an expected part of the observed buggy run.
pass
def _body(self) -> dict[str, object]:
length = int(self.headers.get("content-length") or 0)
if not length:
return {}
value = json.loads(self.rfile.read(length))
return value if isinstance(value, dict) else {}
def do_GET(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
parsed = urlparse(self.path)
if parsed.path == "/health":
self._json(200, {"ok": True})
return
if parsed.path.startswith("/orders/"):
order_id = parsed.path.rsplit("/", 1)[-1]
self._json(200, {"order_id": order_id, "status": "paid", "sku": "SKU-42"})
return
if parsed.path.startswith("/inventory/"):
sku = parsed.path.rsplit("/", 1)[-1]
degraded = parse_qs(parsed.query).get("degraded", ["0"])[0] == "1"
if degraded:
self._json(200, {"sku": sku, "stock": 12, "source": "cache", "degraded": True})
else:
# Longer than the campaign's real client deadline.
time.sleep(0.8)
self._json(200, {"sku": sku, "stock": 12, "source": "origin", "degraded": False})
return
self._json(404, {"error": "not_found"})
def do_POST(self) -> None: # noqa: N802 - BaseHTTPRequestHandler contract
body = self._body()
if self.path == "/refund/eligibility":
self._json(200, {"order_id": body.get("order_id"), "eligible": True})
return
if self.path == "/refund/process":
self._json(200, {"order_id": body.get("order_id"), "refund_id": "RF-LIVE-1"})
return
if self.path == "/notifications":
self._json(200, {"sent": True, "status": body.get("status")})
return
self._json(404, {"error": "not_found"})
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--port", type=int, required=True)
args = parser.parse_args()
server = ThreadingHTTPServer(("127.0.0.1", args.port), Handler)
print(json.dumps({"listening": args.port}), flush=True)
server.serve_forever()
if __name__ == "__main__":
main()
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/live_replays.json
+529
View File
@@ -0,0 +1,529 @@
{
"results": [
{
"test": {
"test_id": "TC-R1-HTTP-RF-001-T2",
"trajectory_id": "HTTP-RF-001",
"focus_turn": 2,
"description": "Refund flow must include verify_refund_eligibility before process_refund (R1)",
"assertion": {
"type": "step_present",
"params": {
"tool": "verify_refund_eligibility"
}
}
},
"buggy": {
"passed": false,
"detail": "verify_refund_eligibility calls=0",
"trajectory": {
"trajectory_id": "HTTP-RF-001::buggy",
"source_trajectory_id": "HTTP-RF-001",
"implementation": "buggy",
"task_input": {
"intent": "refund",
"order_id": "ORD-58-A"
},
"final_status": "success",
"turns": [
{
"index": 0,
"role": "user",
"content": "{\"intent\": \"refund\", \"order_id\": \"ORD-58-A\"}"
},
{
"index": 1,
"role": "tool",
"module": "http_order_system",
"tool": "query_order",
"method": "GET",
"path": "/orders/ORD-58-A",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 1.833,
"response": {
"order_id": "ORD-58-A",
"status": "paid",
"sku": "SKU-42"
}
},
{
"index": 2,
"role": "tool",
"module": "http_order_system",
"tool": "process_refund",
"method": "POST",
"path": "/refund/process",
"request": {
"order_id": "ORD-58-A"
},
"http_status": 200,
"status": "success",
"latency_ms": 0.833,
"response": {
"order_id": "ORD-58-A",
"refund_id": "RF-LIVE-1"
}
},
{
"index": 3,
"role": "tool",
"module": "http_order_system",
"tool": "notify_user",
"method": "POST",
"path": "/notifications",
"request": {
"status": "success"
},
"http_status": 200,
"status": "success",
"latency_ms": 0.678,
"response": {
"sent": true,
"status": "success"
}
}
]
}
},
"fixed": {
"passed": true,
"detail": "verify_refund_eligibility calls=1",
"trajectory": {
"trajectory_id": "HTTP-RF-001::fixed",
"source_trajectory_id": "HTTP-RF-001",
"implementation": "fixed",
"task_input": {
"intent": "refund",
"order_id": "ORD-58-A"
},
"final_status": "success",
"turns": [
{
"index": 0,
"role": "user",
"content": "{\"intent\": \"refund\", \"order_id\": \"ORD-58-A\"}"
},
{
"index": 1,
"role": "tool",
"module": "http_order_system",
"tool": "query_order",
"method": "GET",
"path": "/orders/ORD-58-A",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 0.498,
"response": {
"order_id": "ORD-58-A",
"status": "paid",
"sku": "SKU-42"
}
},
{
"index": 2,
"role": "tool",
"module": "http_order_system",
"tool": "verify_refund_eligibility",
"method": "POST",
"path": "/refund/eligibility",
"request": {
"order_id": "ORD-58-A"
},
"http_status": 200,
"status": "success",
"latency_ms": 0.379,
"response": {
"order_id": "ORD-58-A",
"eligible": true
}
},
{
"index": 3,
"role": "tool",
"module": "http_order_system",
"tool": "process_refund",
"method": "POST",
"path": "/refund/process",
"request": {
"order_id": "ORD-58-A"
},
"http_status": 200,
"status": "success",
"latency_ms": 0.37,
"response": {
"order_id": "ORD-58-A",
"refund_id": "RF-LIVE-1"
}
},
{
"index": 4,
"role": "tool",
"module": "http_order_system",
"tool": "notify_user",
"method": "POST",
"path": "/notifications",
"request": {
"status": "success"
},
"http_status": 200,
"status": "success",
"latency_ms": 0.315,
"response": {
"sent": true,
"status": "success"
}
}
]
}
}
},
{
"test": {
"test_id": "TC-R2-LATENCY-HTTP-INV-001-T2",
"trajectory_id": "HTTP-INV-001",
"focus_turn": 2,
"description": "check_stock must complete within 250 ms (R2)",
"assertion": {
"type": "latency_under",
"params": {
"tool": "check_stock",
"threshold_ms": 250
}
}
},
"buggy": {
"passed": false,
"detail": "check_stock max_latency_ms=350.383, threshold_ms=250.000",
"trajectory": {
"trajectory_id": "HTTP-INV-001::buggy",
"source_trajectory_id": "HTTP-INV-001",
"implementation": "buggy",
"task_input": {
"intent": "order_status",
"order_id": "ORD-58-B",
"sku": "SKU-42"
},
"final_status": "failed",
"turns": [
{
"index": 0,
"role": "user",
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
},
{
"index": 1,
"role": "tool",
"module": "http_order_system",
"tool": "query_order",
"method": "GET",
"path": "/orders/ORD-58-B",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 0.308,
"response": {
"order_id": "ORD-58-B",
"status": "paid",
"sku": "SKU-42"
}
},
{
"index": 2,
"role": "tool",
"module": "http_order_system",
"tool": "check_stock",
"method": "GET",
"path": "/inventory/SKU-42",
"request": null,
"http_status": null,
"status": "error",
"latency_ms": 350.383,
"error": "TimeoutError: timed out"
},
{
"index": 3,
"role": "tool",
"module": "http_order_system",
"tool": "notify_user",
"method": "POST",
"path": "/notifications",
"request": {
"status": "failed"
},
"http_status": 200,
"status": "success",
"latency_ms": 1.481,
"response": {
"sent": true,
"status": "failed"
}
}
]
}
},
"fixed": {
"passed": true,
"detail": "check_stock max_latency_ms=101.234, threshold_ms=250.000",
"trajectory": {
"trajectory_id": "HTTP-INV-001::fixed",
"source_trajectory_id": "HTTP-INV-001",
"implementation": "fixed",
"task_input": {
"intent": "order_status",
"order_id": "ORD-58-B",
"sku": "SKU-42"
},
"final_status": "success",
"turns": [
{
"index": 0,
"role": "user",
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
},
{
"index": 1,
"role": "tool",
"module": "http_order_system",
"tool": "query_order",
"method": "GET",
"path": "/orders/ORD-58-B",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 0.843,
"response": {
"order_id": "ORD-58-B",
"status": "paid",
"sku": "SKU-42"
}
},
{
"index": 2,
"role": "tool",
"module": "http_order_system",
"tool": "check_stock",
"method": "GET",
"path": "/inventory/SKU-42",
"request": null,
"http_status": null,
"status": "error",
"latency_ms": 101.234,
"error": "TimeoutError: timed out"
},
{
"index": 3,
"role": "tool",
"module": "http_order_system",
"tool": "check_stock",
"method": "GET",
"path": "/inventory/SKU-42?degraded=1",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 1.289,
"response": {
"sku": "SKU-42",
"stock": 12,
"source": "cache",
"degraded": true
}
},
{
"index": 4,
"role": "tool",
"module": "http_order_system",
"tool": "notify_user",
"method": "POST",
"path": "/notifications",
"request": {
"status": "success"
},
"http_status": 200,
"status": "success",
"latency_ms": 0.78,
"response": {
"sent": true,
"status": "success"
}
}
]
}
}
},
{
"test": {
"test_id": "TC-R2-STATUS-HTTP-INV-001-T2",
"trajectory_id": "HTTP-INV-001",
"focus_turn": 2,
"description": "check_stock origin timeout must use degraded cache route, resulting in successful flow (R2)",
"assertion": {
"type": "final_status_is",
"params": {
"value": "success"
}
}
},
"buggy": {
"passed": false,
"detail": "final_status=failed, expected=success",
"trajectory": {
"trajectory_id": "HTTP-INV-001::buggy",
"source_trajectory_id": "HTTP-INV-001",
"implementation": "buggy",
"task_input": {
"intent": "order_status",
"order_id": "ORD-58-B",
"sku": "SKU-42"
},
"final_status": "failed",
"turns": [
{
"index": 0,
"role": "user",
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
},
{
"index": 1,
"role": "tool",
"module": "http_order_system",
"tool": "query_order",
"method": "GET",
"path": "/orders/ORD-58-B",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 0.608,
"response": {
"order_id": "ORD-58-B",
"status": "paid",
"sku": "SKU-42"
}
},
{
"index": 2,
"role": "tool",
"module": "http_order_system",
"tool": "check_stock",
"method": "GET",
"path": "/inventory/SKU-42",
"request": null,
"http_status": null,
"status": "error",
"latency_ms": 355.612,
"error": "TimeoutError: timed out"
},
{
"index": 3,
"role": "tool",
"module": "http_order_system",
"tool": "notify_user",
"method": "POST",
"path": "/notifications",
"request": {
"status": "failed"
},
"http_status": 200,
"status": "success",
"latency_ms": 1.19,
"response": {
"sent": true,
"status": "failed"
}
}
]
}
},
"fixed": {
"passed": true,
"detail": "final_status=success, expected=success",
"trajectory": {
"trajectory_id": "HTTP-INV-001::fixed",
"source_trajectory_id": "HTTP-INV-001",
"implementation": "fixed",
"task_input": {
"intent": "order_status",
"order_id": "ORD-58-B",
"sku": "SKU-42"
},
"final_status": "success",
"turns": [
{
"index": 0,
"role": "user",
"content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"
},
{
"index": 1,
"role": "tool",
"module": "http_order_system",
"tool": "query_order",
"method": "GET",
"path": "/orders/ORD-58-B",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 0.895,
"response": {
"order_id": "ORD-58-B",
"status": "paid",
"sku": "SKU-42"
}
},
{
"index": 2,
"role": "tool",
"module": "http_order_system",
"tool": "check_stock",
"method": "GET",
"path": "/inventory/SKU-42",
"request": null,
"http_status": null,
"status": "error",
"latency_ms": 102.492,
"error": "TimeoutError: timed out"
},
{
"index": 3,
"role": "tool",
"module": "http_order_system",
"tool": "check_stock",
"method": "GET",
"path": "/inventory/SKU-42?degraded=1",
"request": null,
"http_status": 200,
"status": "success",
"latency_ms": 1.493,
"response": {
"sku": "SKU-42",
"stock": 12,
"source": "cache",
"degraded": true
}
},
{
"index": 4,
"role": "tool",
"module": "http_order_system",
"tool": "notify_user",
"method": "POST",
"path": "/notifications",
"request": {
"status": "success"
},
"http_status": 200,
"status": "success",
"latency_ms": 0.843,
"response": {
"sent": true,
"status": "success"
}
}
]
}
}
}
]
}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/manifest.json
+81
View File
@@ -0,0 +1,81 @@
{
"schema_version": "1.0",
"experiment": "5-8",
"run_id": "exp5-8-live-http-mcp-20260730-053403",
"generated_at_utc": "2026-07-29T21:37:08Z",
"provider": "ark",
"model": "doubao-seed-1-6-250615",
"service": {
"kind": "real local HTTP subprocess",
"base_url": "http://127.0.0.1:56938",
"pid": 96265
},
"source_trajectory_ids": [
"HTTP-INV-001",
"HTTP-RF-001"
],
"model_call_count": 2,
"github_issue_url": "https://github.com/bojieli/ai-agent-book/issues/502",
"gates": {
"real_local_http_trajectories": true,
"measured_latency_and_raw_http": true,
"live_model_diagnosis": true,
"diagnosis_references_trajectories_and_turns": true,
"live_model_generated_executable_tests": true,
"buggy_failures_reproduced": true,
"fixed_system_passes": true,
"official_github_mcp_issue_created": true,
"raw_provider_receipts_complete": true
},
"artifacts": {
"PRD.md": {
"sha256": "d21f65b3f5243f79b6ab46525945b8e882a6646d75177062cb4d976192369353",
"bytes": 448
},
"architecture.md": {
"sha256": "e76c20c6ac826325db0c7b35204b0ce7235973dc33e3b62220aae607a60f49e4",
"bytes": 422
},
"diagnosis.json": {
"sha256": "4232addd060d6ace66814d53625d6b08b8511e77c34253c3d9e1529199d4b570",
"bytes": 1334
},
"github_mcp_receipt.json": {
"sha256": "30c42f32006eaf0fda5350747ce2bbf44d61af79a6020a62ab12bfb28836976e",
"bytes": 1882
},
"http_service.py": {
"sha256": "bbe0170e342a34fb70dc2477fede68cd393e1fd9a6d98140974a61677413c997",
"bytes": 3701
},
"live_replays.json": {
"sha256": "ef8401bed39405e0398d47a2b3553d50c50d1199b359873afa180b60153b29a8",
"bytes": 16057
},
"production_trajectories.jsonl": {
"sha256": "9d45f81e64ff8b6ccaa109dbdc838bb016832bcae14633fc86a9a917fccb0e6e",
"bytes": 2300
},
"provider_receipts.checkpoint.json": {
"sha256": "20e466f383f807d9ad1d6e0a1a699733f7e7e4da2d459a3add1a976578fb2bd0",
"bytes": 12171
},
"provider_receipts.json": {
"sha256": "20e466f383f807d9ad1d6e0a1a699733f7e7e4da2d459a3add1a976578fb2bd0",
"bytes": 12171
},
"regression_tests.json": {
"sha256": "642d9037754d16665be684f69eb6decd74e8db25181b214bce806e34ceeb7c3a",
"bytes": 1091
},
"service.stderr.log": {
"sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"bytes": 0
},
"service.stdout.jsonl": {
"sha256": "776fe22d507f08f0b56e017599777a60c0165a70e57556e34b3540a27b25fe5a",
"bytes": 2987
}
},
"official_complete": true
}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/production_trajectories.jsonl
+2
View File
@@ -0,0 +1,2 @@
{"trajectory_id": "HTTP-RF-001::buggy", "source_trajectory_id": "HTTP-RF-001", "implementation": "buggy", "task_input": {"intent": "refund", "order_id": "ORD-58-A"}, "final_status": "success", "turns": [{"index": 0, "role": "user", "content": "{\"intent\": \"refund\", \"order_id\": \"ORD-58-A\"}"}, {"index": 1, "role": "tool", "module": "http_order_system", "tool": "query_order", "method": "GET", "path": "/orders/ORD-58-A", "request": null, "http_status": 200, "status": "success", "latency_ms": 0.463, "response": {"order_id": "ORD-58-A", "status": "paid", "sku": "SKU-42"}}, {"index": 2, "role": "tool", "module": "http_order_system", "tool": "process_refund", "method": "POST", "path": "/refund/process", "request": {"order_id": "ORD-58-A"}, "http_status": 200, "status": "success", "latency_ms": 0.393, "response": {"order_id": "ORD-58-A", "refund_id": "RF-LIVE-1"}}, {"index": 3, "role": "tool", "module": "http_order_system", "tool": "notify_user", "method": "POST", "path": "/notifications", "request": {"status": "success"}, "http_status": 200, "status": "success", "latency_ms": 0.444, "response": {"sent": true, "status": "success"}}]}
{"trajectory_id": "HTTP-INV-001::buggy", "source_trajectory_id": "HTTP-INV-001", "implementation": "buggy", "task_input": {"intent": "order_status", "order_id": "ORD-58-B", "sku": "SKU-42"}, "final_status": "failed", "turns": [{"index": 0, "role": "user", "content": "{\"intent\": \"order_status\", \"order_id\": \"ORD-58-B\", \"sku\": \"SKU-42\"}"}, {"index": 1, "role": "tool", "module": "http_order_system", "tool": "query_order", "method": "GET", "path": "/orders/ORD-58-B", "request": null, "http_status": 200, "status": "success", "latency_ms": 0.391, "response": {"order_id": "ORD-58-B", "status": "paid", "sku": "SKU-42"}}, {"index": 2, "role": "tool", "module": "http_order_system", "tool": "check_stock", "method": "GET", "path": "/inventory/SKU-42", "request": null, "http_status": null, "status": "error", "latency_ms": 350.414, "error": "TimeoutError: timed out"}, {"index": 3, "role": "tool", "module": "http_order_system", "tool": "notify_user", "method": "POST", "path": "/notifications", "request": {"status": "failed"}, "http_status": 200, "status": "success", "latency_ms": 2.443, "response": {"sent": true, "status": "failed"}}]}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/provider_receipts.checkpoint.json
+72
View File
File diff suppressed because one or more lines are too long
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/provider_receipts.json
+72
View File
File diff suppressed because one or more lines are too long
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/regression_tests.json
+41
View File
@@ -0,0 +1,41 @@
{
"test_cases": [
{
"test_id": "TC-R1-HTTP-RF-001-T2",
"trajectory_id": "HTTP-RF-001",
"focus_turn": 2,
"description": "Refund flow must include verify_refund_eligibility before process_refund (R1)",
"assertion": {
"type": "step_present",
"params": {
"tool": "verify_refund_eligibility"
}
}
},
{
"test_id": "TC-R2-LATENCY-HTTP-INV-001-T2",
"trajectory_id": "HTTP-INV-001",
"focus_turn": 2,
"description": "check_stock must complete within 250 ms (R2)",
"assertion": {
"type": "latency_under",
"params": {
"tool": "check_stock",
"threshold_ms": 250
}
}
},
{
"test_id": "TC-R2-STATUS-HTTP-INV-001-T2",
"trajectory_id": "HTTP-INV-001",
"focus_turn": 2,
"description": "check_stock origin timeout must use degraded cache route, resulting in successful flow (R2)",
"assertion": {
"type": "final_status_is",
"params": {
"value": "success"
}
}
}
]
}
chapter5/log-diagnosis/validation/runs/exp5-8-live-http-mcp-20260730-053403/service.stdout.jsonl
+29
View File
@@ -0,0 +1,29 @@
{"listening": 56938}
{"client": "127.0.0.1", "message": "\"GET /health HTTP/1.1\" 200 -", "time": 1785360843.868107}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-A HTTP/1.1\" 200 -", "time": 1785360843.8689358}
{"client": "127.0.0.1", "message": "\"POST /refund/process HTTP/1.1\" 200 -", "time": 1785360843.869437}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785360843.8698602}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785360843.8703701}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785360844.223199}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785360844.6740758}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-A HTTP/1.1\" 200 -", "time": 1785361025.351637}
{"client": "127.0.0.1", "message": "\"POST /refund/process HTTP/1.1\" 200 -", "time": 1785361025.353014}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.3537788}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-A HTTP/1.1\" 200 -", "time": 1785361025.3544052}
{"client": "127.0.0.1", "message": "\"POST /refund/eligibility HTTP/1.1\" 200 -", "time": 1785361025.354883}
{"client": "127.0.0.1", "message": "\"POST /refund/process HTTP/1.1\" 200 -", "time": 1785361025.3552582}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.355627}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361025.355964}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.707724}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361025.708798}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42?degraded=1 HTTP/1.1\" 200 -", "time": 1785361025.811397}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361025.812402}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361025.813149}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.157606}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361026.169961}
{"client": "127.0.0.1", "message": "\"GET /orders/ORD-58-B HTTP/1.1\" 200 -", "time": 1785361026.1710122}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42?degraded=1 HTTP/1.1\" 200 -", "time": 1785361026.275081}
{"client": "127.0.0.1", "message": "\"POST /notifications HTTP/1.1\" 200 -", "time": 1785361026.276181}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.5155609}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.6163042}
{"client": "127.0.0.1", "message": "\"GET /inventory/SKU-42 HTTP/1.1\" 200 -", "time": 1785361026.972992}