ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,3 @@
|
||||
.env
|
||||
__pycache__/
|
||||
*.pyc
|
||||
@@ -0,0 +1,413 @@
|
||||
# Experiment 5-3: Codified Rules for Small Models / 实验 5-3：小模型通过代码化知识提升执行规则的准确性
|
||||
|
||||
> Companion lab for *AI Agents in Depth*, Chapter 5 — τ-bench-style airline customer service: codified refund policy as CODE guard vs pure natural-language policy.
|
||||
> 《深入理解 AI Agent》第 5 章（实验 5-3）：τ-bench 航空客服对照——把退款规则从提示词搬进代码/工具。
|
||||
|
||||
← [Chapter 5 index / 返回第 5 章目录](../README.md)
|
||||
|
||||
## Code map
|
||||
|
||||
- **Run first:** `python demo.py --selftest` (offline policy/guard smoke); model-backed runs require the configured provider.
|
||||
- **Start here:** `airline_env.py::AirlineEnv` owns database truth and `is_refundable`; `agent.py::run_agent` drives one conversation.
|
||||
- **Core behavior:** `agent.py::_dispatch` separates control and codified tools; `demo.py::judge` scores the final environment state.
|
||||
- **State / protocol:** `Reservation`, `tasks.py::TASKS`, checklist records, tool transcript and per-arm result JSON.
|
||||
- **Verifier:** `test_campaign.py`, deterministic `judge`, and the validation manifest/paired analysis.
|
||||
- **Experiment variable:** `control` versus `codified`, model/provider, task subset and checklist parameters.
|
||||
- **Skip on first pass:** provider retry/serialization code, checkpoint plumbing and report formatting.
|
||||
|
||||
## Formal manuscript result (canonical)
|
||||
|
||||
The canonical campaign ran local Ollama `qwen3:4b` on all 60 frozen policy
|
||||
cases in both matched arms (120 complete trajectories). The codified arm
|
||||
verified database facts and server time and exposed checklist parameters, but
|
||||
scored 91.7% versus 95.0% for the natural-language control (exact paired
|
||||
p=0.6875). This is a complete **negative** hypothesis result, not evidence of a
|
||||
significant gain. The raw messages, tool transcripts, usage, policy truth, and
|
||||
paired analysis are in
|
||||
[`validation/real_ollama_qwen3_4b_60x2_20260730.json`](validation/real_ollama_qwen3_4b_60x2_20260730.json).
|
||||
|
||||
正式活动用本地 Ollama `qwen3:4b` 完成固定 60 个政策案例的两组配对运行，共 120 条完整
|
||||
轨迹。代码化组确实执行了数据库真值、服务端时钟与 checklist 门禁，但成功率 91.7%，
|
||||
控制组 95.0%，精确配对检验 p=0.6875，未出现显著提升。这是完整而诚实的负结论；
|
||||
下文 8 题示例表只用于解释机制，不能当作正式实验结果。
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
### One-line takeaway
|
||||
|
||||
Same small model, same tasks: only moving business rules from the prompt into **code/tools** raised **task success from 88% to 100%** and **policy violations from 1 to 0**—and tool-side code checks intercept wrong model beliefs in real time. Claim: **codifying business rules as guards lets a small model match large-model bare reliability on complex policy** (run a large-model baseline arm with `--big-model`; see below).
|
||||
|
||||
### Experiment design
|
||||
|
||||
A simplified airline customer-service env: simulated DB truth (flights / bookings / cabin / booking time / flight status) and one **codified refund policy** (`airline_env.is_refundable`) as the sole authority.
|
||||
|
||||
Refund policy (NL and code share the same source of truth):
|
||||
|
||||
- Basic economy (`basic_economy`) is **non-refundable** by default;
|
||||
- Exception 1: full refund within **24 hours** of booking;
|
||||
- Exception 2: full refund if the airline **cancels** or delays **≥ 3 hours** (major delay);
|
||||
- Flexible / business cabin: full refund;
|
||||
- When non-refundable: explain policy and **proactively offer alternatives** (rebook keeping the ticket, travel credit).
|
||||
|
||||
#### Arms
|
||||
|
||||
By default two arms (same small model; only difference is “codified rules or not”); `--big-model` adds a third **large-model bare baseline**:
|
||||
|
||||
| Arm | Model | Codified rules | Role |
|
||||
|---|---|---|---|
|
||||
| A `codified` | Small | ✅ triple guard | Treatment |
|
||||
| B `control` | Small | ❌ pure NL | Control |
|
||||
| C `control` | **Large** | ❌ pure NL | Large-model baseline (`--big-model`, optional) |
|
||||
|
||||
Expected relation **A ≈ C > B**: small model + codified guards (A) matches large bare (C), both clearly beat small bare (B).
|
||||
|
||||
#### Only difference between control and treatment (three-layer guards)
|
||||
|
||||
| | Control `control` | Treatment `codified` |
|
||||
|---|---|---|
|
||||
| ① System prompt (layer 1) | NL policy | Same NL policy |
|
||||
| ② Tool description (layer 2 · checklist) | Minimal, no checklist params | Full policy listed; optional `expected_refundable` / `expected_reason` nudge model to **check before calling** |
|
||||
| ③ Inside tool (layer 3 · gatekeeper) | Naive: cancel → **unconditional refund** | Codified check on **DB truth**: policy facts from DB, server clock, ignore model self-reports; illegal calls **rejected** |
|
||||
|
||||
Both share read-only `get_reservation` (truth; `hours_since_booking` from server clock so the model cannot mis-compute time). The clean isolation is “whether the third layer exists: in-tool codified validation.” Together they are the book’s “triple guarantee”: first two layers reduce mistakes; the third ensures mistakes do not become irreversible loss.
|
||||
|
||||
#### Eval tasks (8: 4 refundable / 4 non-refundable)
|
||||
|
||||
Normal and adversarial boundary cases: flexible fare, 24h boundary (5h / 26h), airline cancel, business cabin, user lies about flexible fare, minor delay (not major), airline schedule change (neither cancel nor ≥3h delay). See `tasks.py`.
|
||||
|
||||
#### Metrics (rule-based, deterministic, reproducible, zero extra LLM cost)
|
||||
|
||||
“State is truth”: after a run, check whether `refund_issued` happened and compare to codified policy truth:
|
||||
|
||||
- **Task success**: refund outcome matches policy truth;
|
||||
- **Policy violations**: `over-refund (should deny but didn’t)` + `under-refund (should refund but didn’t)`;
|
||||
- **Invalid tool calls**: rejected by code checks / unknown booking, etc. (error/rejected);
|
||||
- **Rate of `expected_*` vs DB truth mismatch** (treatment only): quantifies “model self-belief can be wrong,” motivating server-side truth checks.
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 5 environment
|
||||
uv sync --locked --python 3.12 --extra ch5
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch5]"
|
||||
|
||||
cd chapter5/small-model-codified-rules
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
cp env.example .env # set OPENAI_API_KEY (or env vars)
|
||||
# Fallback: if OPENAI_API_KEY unset, OPENROUTER_API_KEY routes to OpenRouter
|
||||
# (small model gpt-5.6-luna is gpt-5.x → prefer OpenRouter openai/gpt-5.6-luna; large baseline same)
|
||||
|
||||
# Offline self-test (no API key): show codified guard logic
|
||||
python demo.py --selftest
|
||||
|
||||
# Default: all 8 cases, control vs treatment (small model both)
|
||||
python demo.py
|
||||
|
||||
# Three-way: add large-model bare arm
|
||||
python demo.py --big-model gpt-5.6-luna
|
||||
```
|
||||
|
||||
#### CLI (`python demo.py --help` for full Chinese help)
|
||||
|
||||
| Flag | Description |
|
||||
|---|---|
|
||||
| `--mode {control,codified,both}` | Which arm(s); default `both` |
|
||||
| `--task ID [ID ...]` | Cases whose `task_id` matches substring, e.g. `--task R009` |
|
||||
| `--small-model NAME` | Small model (default `gpt-5.6-luna`, or `MODEL`) |
|
||||
| `--big-model NAME` | Large baseline (optional; or `BIG_MODEL`) |
|
||||
| `--quick` | First 4 cases only |
|
||||
| `-v, --verbose` | Print each tool call |
|
||||
| `--output PATH` | Write per-case results + summary JSON |
|
||||
| `--selftest` | Offline codified-check demo (no API key) |
|
||||
|
||||
`--selftest` prints policy truth for all cases and contrasts naive tool (unconditional refund → violation when non-refundable) vs codified tool (always DB truth; block non-refundable)—fastest way to understand layer 3.
|
||||
|
||||
### Real results (`gpt-5.6-luna`, reasoning model temperature=1)
|
||||
|
||||
```
|
||||
指标 控制组 实验组
|
||||
--------------------------------------------------------------------
|
||||
任务成功率 7/8 = 88% 8/8 = 100%
|
||||
政策违规次数 1 0
|
||||
无效工具调用次数 0 1 （= 1 次违规被代码拦截）
|
||||
|
||||
[实验组] expected_* 自报值 vs 数据库真值：
|
||||
5 次带 checklist 的取消调用中，1 次与真值不一致 —— 不一致比例 = 20%
|
||||
```
|
||||
|
||||
> **Large-model arm (C)**: the table is the two-arm run shipped with the repo (same small model `gpt-5.6-luna`). Arm C is **run on demand**—`--big-model <your large model>` fills column C live to check **A (small+rules) ≈ C (large bare) > B (small bare)**. No pre-filled C numbers, so they match your model/time.
|
||||
|
||||
> **What is stable vs noisy**: core claim—success 88%→100%, violations 1→0, control’s only violation fixed on trap case `R009`—reproduces every run; secondary metrics (invalid calls 0–1, `expected_*` mismatch 0%–20%, whether R009 is self-identified at params vs rejected after cancel) vary with reasoning choices. Both paths still end non-refund correctly.
|
||||
|
||||
> **Model ↔ harness, two layers of value.** On weak `gpt-4o-mini`: control 6/8, treatment 8/8 (**+2**); on stronger `gpt-5.6-luna`: control 7/8, gap **+1**. Accuracy gains thin as models get stronger (like `code-for-logic` / `code-for-math`). Codified rules also give a **second value that does not vanish with stronger models: determinism and truth backstop**. Even strong models can over-refund on traps like `R009`; “always check DB, never trust model self-report” stably intercepts for 0 violations. Accuracy can be absorbed by better models; **auditability/safety backstops should not be thinned away blindly**.
|
||||
|
||||
Control’s only stable violation is trap case `R009` (model belief ≠ truth):
|
||||
|
||||
| case | Policy truth | Model belief | Control | Treatment |
|
||||
|---|---|---|---|---|
|
||||
| R009 (basic · airline schedule change, not either exception) | Non-refundable | Treats “airline retime = airline cause = refundable” (`expected_refundable=True`) | ❌ Over-refund | ✅ Code rejects (or self-identify at params) → explain + alternatives |
|
||||
|
||||
`gpt-5.6-luna` as a stronger small model already handles routine 24h cases (e.g. R003 at 5h) but can still be **over-generous** on “airline unilateral retime ≠ cancel / major delay”; server truth checks exist for that.
|
||||
|
||||
#### Intercept example (R009)
|
||||
|
||||
```
|
||||
模型 checklist 自报：expected_refundable=True，expected_reason=airline_caused（认为"航司改签=航司原因=可退"）
|
||||
数据库真值 ：refundable=False，原因=non_refundable_basic_economy
|
||||
模型发起取消调用：{'reservation_id':'R009','expected_refundable':True,'expected_reason':'airline_caused'}
|
||||
工具代码化校验返回：status=rejected, reason=policy_violation
|
||||
→ "已按数据库真值校验：该预订不可退款（基础经济票，下单超过 24 小时，且无航司原因）。
|
||||
系统已拦截退款操作。请勿承诺退款，改为向乘客解释政策，并主动提议替代方案（如保留客票改签、申请旅行信用点）。"
|
||||
|
||||
模型最终回复用户（被拦截后自主转向）：
|
||||
"……根据退款政策，基础经济票仅在下单 24 小时内，或航班被取消、重大延误（≥3 小时）时可退款。
|
||||
目前不符合退款条件，因此无法为 R009 办理全额退款。你可以选择保留客票并申请改签到其他可用航班，
|
||||
或咨询是否能够申请旅行信用点……"
|
||||
```
|
||||
|
||||
In **control**, the naive tool refunds—`gpt-5.6-luna` treated unilateral retime as a refund reason. NL-only complex policy is unreliable; **codify rules in tools** so wrong model judgment is still blocked and the dialogue pivots to explanation + alternatives.
|
||||
|
||||
### Two key observations
|
||||
|
||||
1. **“Params as checklist”**: in treatment, preparing `expected_*` is guided by the tool description; many boundary cases (R006 26h, R008 minor delay, R005 user lie) are **self-identified as non-refundable** before any cancel, and the agent explains + offers alternatives instead of refunding.
|
||||
2. **Need for server-side truth**: self-belief is often good but still wrong on traps like R009—this run had **20% (1/5)** `expected_*` vs truth mismatch (other runs 0%–20%). Control trusts the model and turns that into over-refund. Offline: `python demo.py --selftest` injects inverted self-reports to show intercepts deterministically without a key.
|
||||
|
||||
### Files
|
||||
|
||||
- `airline_env.py`: simulated DB, codified `is_refundable`, naive vs codified tools.
|
||||
- `tasks.py`: 8 eval tasks and policy truth.
|
||||
- `agent.py`: OpenAI tool loop; system prompts and tool schemas for both arms (`run_agent` takes `model` for the large baseline).
|
||||
- `demo.py`: arms, eval, rule scoring, N-arm table + mismatch rate + intercept samples; CLI and offline self-test.
|
||||
- `requirements.txt` / `env.example`.
|
||||
|
||||
### Caveats
|
||||
|
||||
- Use `OPENAI_API_KEY` (default small model `gpt-5.6-luna`; `MODEL` / `--small-model`; large baseline `BIG_MODEL` / `--big-model`). Cost is low (~tens of calls per arm × 8 cases).
|
||||
- No key: `python demo.py --selftest`.
|
||||
- Reasoning models (`gpt-5.6-luna` and gpt-5/o series) reject `temperature=0`; code uses `temperature=1`, so secondary metrics can drift slightly while “treatment ≥ control and treatment 8/8 with 0 violations” stays stable.
|
||||
- Server clock fixed at `2026-07-17 12:00` (`airline_env.SERVER_NOW`); all time logic uses it.
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 一句话结论
|
||||
|
||||
同一个小模型、同一批任务，仅仅把"业务规则从提示词搬进代码/工具"，就把
|
||||
**任务成功率从 88% 提升到 100%，政策违规从 1 次降到 0 次**——并能观察到工具内代码
|
||||
校验实时拦截了模型的错误认知。核心主张：**把业务规则代码化为守卫，能让一个小模型在
|
||||
复杂政策执行上追平大模型裸跑**（用 `--big-model` 加跑大模型基线臂即可现场验证，见下文）。
|
||||
|
||||
### 实验设计
|
||||
|
||||
精简航空客服环境：一个模拟"数据库真值"（航班/预订/舱位/下单时间/航班状态），一条
|
||||
**代码化的退款政策**（`airline_env.is_refundable`）作为唯一权威判据。
|
||||
|
||||
退款政策（自然语言 + 代码同源）：
|
||||
- 经济舱基础票（`basic_economy`）默认**不可退款**；
|
||||
- 例外 1：下单 **24 小时内**可全额退款；
|
||||
- 例外 2：航班被**航司取消**或**延误 ≥ 3 小时**（重大延误）可全额退款；
|
||||
- 灵活票 / 商务舱可全额退款；
|
||||
- 不可退款时应解释政策并**主动提议替代方案**（保留客票改签、旅行信用点）。
|
||||
|
||||
#### 对照臂
|
||||
|
||||
默认跑两臂（同一小模型，唯一差异是"是否代码化规则"）；加 `--big-model` 则再加一臂
|
||||
**大模型裸跑基线**，凑成书中所述的三方对照：
|
||||
|
||||
| 臂 | 模型 | 代码化规则 | 角色 |
|
||||
|---|---|---|---|
|
||||
| A `codified` | 小模型 | ✅ 三重保障 | 实验组 |
|
||||
| B `control` | 小模型 | ❌ 纯自然语言 | 控制组 |
|
||||
| C `control` | **大模型** | ❌ 纯自然语言 | 大模型基线（`--big-model`，可选） |
|
||||
|
||||
预期关系 **A ≈ C > B**：小模型 + 代码化守卫（A）追平大模型裸跑（C），且都显著优于
|
||||
小模型裸跑（B）。
|
||||
|
||||
#### 控制组 / 实验组的唯一差异（三层守卫对照）
|
||||
|
||||
| | 控制组 `control` | 实验组 `codified` |
|
||||
|---|---|---|
|
||||
| ① 系统提示（第一层守卫） | 自然语言政策 | 自然语言政策（相同） |
|
||||
| ② 工具描述（第二层守卫·checklist） | 极简、无 checklist 参数 | 列出完整政策，并以可选 `expected_refundable` / `expected_reason` 参数引导模型**调用前逐条核对** |
|
||||
| ③ 工具内部（第三层守卫·守门员） | 天真执行：被调用即取消并**无条件退款** | 基于**数据库真值**代码化校验：政策事实一律查库、时间取服务端时钟、不采信模型自报参数；违规调用直接**拒绝** |
|
||||
|
||||
两组共用只读工具 `get_reservation`（返回真值，`hours_since_booking` 由服务端时钟算好，
|
||||
杜绝模型口算时间出错）。差异被干净地隔离为"是否有第三重保障：工具内代码化校验"。
|
||||
三层守卫合起来即书中"三重保障"：前两层减少错误发生，第三层确保错误不会变成不可逆损失。
|
||||
|
||||
#### 评测任务（8 个，可退 4 / 不可退 4）
|
||||
|
||||
含正常任务与违规边界任务，覆盖：灵活票、24h 边界（5h / 26h）、航司取消、商务舱、
|
||||
用户谎称灵活票、轻微延误（非重大延误）、航司改签时刻（既非取消也非 ≥3h 延误）。
|
||||
见 `tasks.py`。
|
||||
|
||||
#### 指标与判据（规则判据，确定性、可复现、零额外成本）
|
||||
|
||||
"状态即真值"：一次运行结束后直接检查环境里 `refund_issued` 是否发生，与代码化政策
|
||||
真值比对：
|
||||
- **任务成功率**：退款结果是否符合政策真值；
|
||||
- **政策违规次数**：`多退款（该拒不拒）` + `该退不退`，两个方向都算；
|
||||
- **无效工具调用次数**：被代码校验拒绝 / 未知预订等返回 error/rejected 的调用；
|
||||
- **`expected_*` 自报值 vs 数据库真值 不一致比例**（仅实验组）：量化"模型自我认知会
|
||||
出错"，从而验证服务端真值校验的必要性。
|
||||
|
||||
### 运行
|
||||
|
||||
```bash
|
||||
# 在仓库根目录使用统一的第 5 章环境
|
||||
uv sync --locked --python 3.12 --extra ch5
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch5]"
|
||||
|
||||
cd chapter5/small-model-codified-rules
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
cp env.example .env # 填入 OPENAI_API_KEY（也可直接用环境变量）
|
||||
# 通用兜底：未配置 OPENAI_API_KEY 时，设置 OPENROUTER_API_KEY 即自动改走 OpenRouter
|
||||
#（小模型 gpt-5.6-luna 属 gpt-5.x，代码会自动优先走 OpenRouter：openai/gpt-5.6-luna；大模型基线同理）
|
||||
|
||||
# 离线自检（无需 API Key）：直接看代码化守卫的校验逻辑
|
||||
python demo.py --selftest
|
||||
|
||||
# 默认：跑全部 8 个 case，控制组 vs 实验组（均用小模型）
|
||||
python demo.py
|
||||
|
||||
# 三方对照：加跑大模型基线臂，验证"小模型+规则 ≈ 大模型裸跑"
|
||||
python demo.py --big-model gpt-5.6-luna
|
||||
```
|
||||
|
||||
#### 命令行参数（`python demo.py --help` 看完整中文帮助）
|
||||
|
||||
| 参数 | 说明 |
|
||||
|---|---|
|
||||
| `--mode {control,codified,both}` | 跑哪一组：不带/带 代码化规则，或两组都跑（默认 `both`） |
|
||||
| `--task ID [ID ...]` | 只跑 `task_id` 匹配子串的 case，如 `--task R009` 直取核心拦截样例 |
|
||||
| `--small-model NAME` | 小模型名（默认 `gpt-5.6-luna`，或用环境变量 `MODEL`） |
|
||||
| `--big-model NAME` | 大模型基线名（可选；给定后加跑第三臂，或用环境变量 `BIG_MODEL`） |
|
||||
| `--quick` | 只跑前 4 个 case（省钱快看） |
|
||||
| `-v, --verbose` | 打印每步工具调用 |
|
||||
| `--output PATH` | 把逐 case 结果与汇总指标写入 JSON |
|
||||
| `--selftest` | 离线演示代码化校验逻辑（无需 API Key） |
|
||||
|
||||
`--selftest` 会对全部 case 打印政策真值，并对比"天真工具（无条件退款、不可退时即违规）"
|
||||
与"代码化工具（一律以数据库真值裁决、不可退一律拦截）"，是理解第三层守卫最快的方式。
|
||||
|
||||
### 真实运行结果（`gpt-5.6-luna`，推理模型 temperature=1）
|
||||
|
||||
```
|
||||
指标 控制组 实验组
|
||||
--------------------------------------------------------------------
|
||||
任务成功率 7/8 = 88% 8/8 = 100%
|
||||
政策违规次数 1 0
|
||||
无效工具调用次数 0 1 （= 1 次违规被代码拦截）
|
||||
|
||||
[实验组] expected_* 自报值 vs 数据库真值：
|
||||
5 次带 checklist 的取消调用中，1 次与真值不一致 —— 不一致比例 = 20%
|
||||
```
|
||||
|
||||
> **关于大模型基线臂（C）**：上表是本仓库随附的真实两臂运行（同一小模型 `gpt-5.6-luna`）。
|
||||
> 第三臂"大模型裸跑"是**按需自跑**的——加 `--big-model <你的大模型>` 即可现场得到 C 列
|
||||
> 成功率填进对比表，验证 **A（小模型+规则）≈ C（大模型裸跑）> B（小模型裸跑）**。
|
||||
> 这里不预填 C 的具体数字，以免与你实际使用的大模型/时刻不符。
|
||||
|
||||
> **哪些数字稳定、哪些会波动**：核心结论——「任务成功率 88%→100%、政策违规 1→0，
|
||||
> 且控制组唯一的违规固定落在陷阱 case `R009`」——每次运行都稳定复现；而次级指标
|
||||
> （实验组无效工具调用次数 0~1、`expected_*` 不一致比例 0%~20%、以及 R009 在实验组
|
||||
> 究竟是"参数阶段就被模型自我识别为不可退"还是"发起取消后被代码守卫拦截"）取决于
|
||||
> 推理模型每次的选择，会小幅波动，属正常现象——两条路径都稳定导向正确的不退款结果。
|
||||
|
||||
> **模型 ↔ 脚手架此消彼长，但这里的脚手架有「两层价值」。** 本实验在强弱两个模型上都实测过：
|
||||
> 较弱模型 `gpt-4o-mini` 控制组 6/8、实验组 8/8，代码化规则把成功率拉开 **+2 题**；换成更强的 `gpt-5.6-luna`，
|
||||
> 控制组自己就升到 7/8，差距收窄到 **+1 题**。可见**准确率**这层收益确实随模型变强而变薄——这与 `code-for-logic`、
|
||||
> `code-for-math` 的规律一致。但代码化规则还提供了**不随模型变强而消失**的第二层价值：**确定性与真值兜底**。
|
||||
> 即便强模型，仍会在 `R009` 这类政策陷阱上"该拒不拒"，而"一律查库校验、不采信模型自报"能稳定拦截、保持 0 违规。
|
||||
> 换言之：靠模型能力能补的部分（准确率）会被更强的模型逐步抹平，靠脚手架才能保证的部分（确定性、可审计、安全兜底）
|
||||
> 不会——**这正是判断"哪些脚手架可以随模型升级而变薄、哪些必须保留"的关键**。
|
||||
|
||||
控制组唯一的违规稳定发生在**模型认知与真值不符**的陷阱 case `R009`，形成清晰的因果链：
|
||||
|
||||
| case | 政策真值 | 模型认知 | 控制组结果 | 实验组结果 |
|
||||
|---|---|---|---|---|
|
||||
| R009（基础票·航司改签时刻，不属两条例外） | 不可退 | 误当"航司改签=航司原因=可退"（`expected_refundable=True`） | ❌ 多退款 | ✅ 代码校验拦截（或参数阶段自我识别），转为解释+提议替代 |
|
||||
|
||||
`gpt-5.6-luna` 作为较强的小模型，在 24h 边界（如 R003 下单 5h）等常规判断上已能自行答对，
|
||||
但面对"航司单方面改签 ≠ 航司取消/重大延误"这类政策细节仍会**过度慷慨、该拒不拒**；服务端
|
||||
真值校验（政策事实一律查库、不采信模型自报参数）正是为兜住这类认知错误而设。
|
||||
|
||||
#### 代码化校验拦截实例（R009）
|
||||
|
||||
```
|
||||
模型 checklist 自报：expected_refundable=True，expected_reason=airline_caused（认为"航司改签=航司原因=可退"）
|
||||
数据库真值 ：refundable=False，原因=non_refundable_basic_economy
|
||||
模型发起取消调用：{'reservation_id':'R009','expected_refundable':True,'expected_reason':'airline_caused'}
|
||||
工具代码化校验返回：status=rejected, reason=policy_violation
|
||||
→ "已按数据库真值校验：该预订不可退款（基础经济票，下单超过 24 小时，且无航司原因）。
|
||||
系统已拦截退款操作。请勿承诺退款，改为向乘客解释政策，并主动提议替代方案（如保留客票改签、申请旅行信用点）。"
|
||||
|
||||
模型最终回复用户（被拦截后自主转向）：
|
||||
"……根据退款政策，基础经济票仅在下单 24 小时内，或航班被取消、重大延误（≥3 小时）时可退款。
|
||||
目前不符合退款条件，因此无法为 R009 办理全额退款。你可以选择保留客票并申请改签到其他可用航班，
|
||||
或咨询是否能够申请旅行信用点……"
|
||||
```
|
||||
|
||||
同一个 case 在**控制组**里，天真工具直接执行了退款——`gpt-5.6-luna` 把"航司单方面改签"
|
||||
当成了退款理由。可见：靠模型自然语言推理执行复杂政策并不可靠；把规则**代码化**到工具内，
|
||||
即使模型判断错误也能被真值兜底拦截，并顺势转为向用户解释与提议替代方案。
|
||||
|
||||
### 观察到的两个关键现象（对应实验目标）
|
||||
|
||||
1. **"参数即 checklist"**：实验组里，模型在**准备 `expected_*` 参数**时就被工具描述的
|
||||
逐条政策引导，多数违规边界（R006 26h、R008 轻微延误、R005 用户谎称）在参数阶段就被
|
||||
模型**自主识别为不可退**，直接向用户解释并提议替代方案，根本没走到退款。
|
||||
2. **服务端真值校验的必要性**：`gpt-5.6-luna` 的自我认知已相当准，但仍会在 R009 这类陷阱上出错——
|
||||
本次运行 `expected_*` 自报值与真值有 **20%（1/5）** 不一致（不同运行在 0%~20% 间波动）；若像控制组那样
|
||||
信任模型自报/自行判断，这个认知错误就会直接变成违规操作（R009 多退款）。想在无 Key 环境下确定性地
|
||||
复现"守卫拦截"，可跑 `python demo.py --selftest`（对每个 case 灌入与真值相反的自报值，演示一律被拦截）。
|
||||
|
||||
### 文件说明
|
||||
|
||||
- `airline_env.py`：模拟数据库、代码化退款政策 `is_refundable`、两组的工具实现（天真 / 代码化校验）。
|
||||
- `tasks.py`：8 个评测任务及其政策真值。
|
||||
- `agent.py`：OpenAI 工具调用循环，两组的系统提示与工具 schema（`run_agent` 支持 `model` 形参，供大模型基线臂复用控制组逻辑）。
|
||||
- `demo.py`：组装对照臂、跑评测、规则判据评分、打印 N 臂指标对比表 + 不一致比例 + 拦截实例；含 CLI（`--mode/--task/--small-model/--big-model/--output/--selftest`）与离线自检。
|
||||
- `requirements.txt` / `env.example`。
|
||||
|
||||
### 注意事项
|
||||
|
||||
- 只用 `OPENAI_API_KEY`（默认小模型 `gpt-5.6-luna`，可用 `MODEL` / `--small-model` 覆盖；
|
||||
大模型基线用 `BIG_MODEL` / `--big-model`）。成本极低（每臂 8 个 case，约几十次调用）。
|
||||
- 想在无 Key 环境下理解代码化守卫，直接 `python demo.py --selftest`。
|
||||
- 推理模型（`gpt-5.6-luna` 等 gpt-5/o 系列）不接受 `temperature=0`，代码会自动改用 `temperature=1`，
|
||||
故次级指标（无效工具调用数、`expected_*` 不一致比例）会小幅波动，个别 case 走的路径偶有出入属正常，
|
||||
但"实验组 ≥ 控制组、且实验组 8/8 无违规"的结论稳定成立。
|
||||
- 服务端时钟固定为 `2026-07-17 12:00`（`airline_env.SERVER_NOW`），所有时间判断以它为准。
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Prefer `--selftest` without a key; use `--big-model` for three-way comparison. / 无 Key 先 `--selftest`；三方对照用 `--big-model`。
|
||||
- Commands/code/paths/env vars are identical in both language sections. / 命令、代码、路径与环境变量在中英文两侧保持一致。
|
||||
@@ -0,0 +1,292 @@
|
||||
"""
|
||||
航空客服 Agent（实验 5-3）
|
||||
|
||||
两种模式：
|
||||
- control（控制组）：系统提示只有自然语言政策；工具描述极简、无 expected_* 参数；
|
||||
工具内部不做任何校验（天真执行）。政策是否被遵守完全靠模型自身推理。
|
||||
- codified（实验组）：三重保障——
|
||||
(1) 系统提示保留同样的自然语言政策；
|
||||
(2) 工具描述列出完整政策，并以可选 expected_* 参数引导模型"调用前逐条核对"；
|
||||
(3) 工具内部基于数据库真值做代码化校验，可拒绝违规操作。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
from airline_env import AirlineEnv
|
||||
|
||||
|
||||
MODEL = os.environ.get("MODEL", "qwen3:4b")
|
||||
MAX_TURNS = 6
|
||||
|
||||
# --- 通用 OpenRouter 兜底 ---
|
||||
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
|
||||
|
||||
def _map_to_openrouter_model(model: str) -> str:
|
||||
"""把直连模型名映射为 OpenRouter 上的 id（非可映射 id 统一兜底到当前廉价旗舰）。"""
|
||||
if not model or "/" in model:
|
||||
return model or "openai/gpt-5.6-luna"
|
||||
m = model.lower()
|
||||
if m.startswith(("gpt-", "o1", "o3", "o4")):
|
||||
return "openai/" + model
|
||||
if m.startswith("claude"):
|
||||
if "haiku" in m:
|
||||
return "anthropic/claude-haiku-4.5"
|
||||
if "sonnet" in m:
|
||||
return "anthropic/claude-sonnet-4.6"
|
||||
return "anthropic/claude-opus-4.8"
|
||||
if m.startswith("gemini"):
|
||||
return "google/" + model
|
||||
return "openai/gpt-5.6-luna"
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 自然语言政策（两组共享，放进系统提示）
|
||||
# ---------------------------------------------------------------------------
|
||||
NL_POLICY = """你是 SkyWing 航空公司的客服 Agent，可以帮乘客查询和取消预订。
|
||||
|
||||
【退款政策（自然语言）】
|
||||
- 经济舱基础票（basic_economy）默认不可退款。
|
||||
- 例外 1：下单后 24 小时内，可全额退款。
|
||||
- 例外 2：航班被航司取消，或发生重大延误（延误 ≥ 3 小时），可全额退款。
|
||||
- 经济舱灵活票（economy_flex）与商务舱（business）可全额退款。
|
||||
- 若不可退款：应礼貌解释政策，并主动提议替代方案（如保留客票改签、申请旅行信用点），
|
||||
绝不能给用户退款。
|
||||
|
||||
请先判断能否退款，再决定是否调用取消/退款工具。乘客自述的信息（舱位、下单时间等）
|
||||
可能不准确，应以系统查询到的预订信息为准。"""
|
||||
|
||||
CONTROL_SYSTEM = NL_POLICY
|
||||
|
||||
CODIFIED_SYSTEM = NL_POLICY + """
|
||||
|
||||
【操作要求】
|
||||
调用 cancel_reservation 前，请先用 get_reservation 查询真实预订信息，逐条核对退款政策，
|
||||
并在 expected_refundable / expected_reason 参数中如实填写你的判断（这是一份调用前 checklist）。
|
||||
系统会以数据库真值为准进行校验：若你的判断与真值不符或存在违规，调用会被拒绝。"""
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 工具 schema
|
||||
# ---------------------------------------------------------------------------
|
||||
GET_RESERVATION_TOOL = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_reservation",
|
||||
"description": "查询预订的详细信息（舱位、下单时间、下单时长、航班状态、价格等，均为系统真值）。",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"reservation_id": {"type": "string", "description": "预订编号，如 R001"},
|
||||
},
|
||||
"required": ["reservation_id"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
CONTROL_CANCEL_TOOL = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "cancel_reservation",
|
||||
"description": "取消一个预订并处理退款。",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"reservation_id": {"type": "string", "description": "预订编号"},
|
||||
},
|
||||
"required": ["reservation_id"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
CODIFIED_CANCEL_TOOL = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "cancel_reservation",
|
||||
"description": (
|
||||
"取消预订并按政策退款。调用前请逐条核对退款政策（这是一份 checklist）：\n"
|
||||
"1) 舱位是否为 basic_economy？非基础经济票可退。\n"
|
||||
"2) 若为基础经济票：下单是否在 24 小时内？（以系统返回的 hours_since_booking 为准）\n"
|
||||
"3) 若为基础经济票：航班是否被航司取消，或延误 ≥ 3 小时（重大延误）？\n"
|
||||
"满足 1 的非基础票、或满足 2/3 例外之一，才可退款。\n"
|
||||
"请在 expected_refundable / expected_reason 中如实填写你的核对结论。"
|
||||
"系统会以数据库真值校验，不可退款的调用将被拒绝。"
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"reservation_id": {"type": "string", "description": "预订编号"},
|
||||
"expected_refundable": {
|
||||
"type": "boolean",
|
||||
"description": "你核对政策后判断该预订是否可退款（checklist 自报值）。",
|
||||
},
|
||||
"expected_reason": {
|
||||
"type": "string",
|
||||
"enum": ["flexible_fare", "within_24h", "airline_caused", "non_refundable_basic_economy"],
|
||||
"description": "你判断可退/不可退的政策依据。",
|
||||
},
|
||||
},
|
||||
"required": ["reservation_id", "expected_refundable", "expected_reason"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def _make_client(model: str | None = None, provider: str = "ollama"):
|
||||
"""构造客户端并解析模型名，含通用 OpenRouter 兜底。返回 (client, resolved_model)。
|
||||
|
||||
- 有 OPENAI_API_KEY：直连；但当 model 是 gpt-5.x 且同时设置了 OPENROUTER_API_KEY
|
||||
时优先走 OpenRouter（直连 gpt-5.6 需组织实名认证）。
|
||||
- 无 OPENAI_API_KEY 但有 OPENROUTER_API_KEY：改走 OpenRouter（模型名自动映射）。
|
||||
"""
|
||||
model = model or MODEL
|
||||
if provider == "ollama":
|
||||
api_key = "ollama"
|
||||
base_url = os.environ.get("OLLAMA_BASE_URL", "http://127.0.0.1:11434/v1")
|
||||
elif provider == "openrouter":
|
||||
api_key = os.environ.get("OPENROUTER_API_KEY")
|
||||
base_url = OPENROUTER_BASE_URL
|
||||
model = _map_to_openrouter_model(model)
|
||||
elif provider == "openai":
|
||||
api_key = os.environ.get("OPENAI_API_KEY")
|
||||
base_url = os.environ.get("OPENAI_BASE_URL")
|
||||
elif provider == "moonshot":
|
||||
api_key = os.environ.get("MOONSHOT_API_KEY")
|
||||
base_url = "https://api.moonshot.cn/v1"
|
||||
elif provider == "ark":
|
||||
api_key = os.environ.get("ARK_API_KEY")
|
||||
base_url = "https://ark.cn-beijing.volces.com/api/v3"
|
||||
else:
|
||||
raise ValueError(f"unsupported provider: {provider}")
|
||||
if not api_key:
|
||||
raise RuntimeError("未设置 OPENAI_API_KEY（或 OPENROUTER_API_KEY 兜底），请参考 env.example 配置。")
|
||||
kw = {"api_key": api_key}
|
||||
if base_url:
|
||||
kw["base_url"] = base_url
|
||||
return OpenAI(**kw), model, provider
|
||||
|
||||
|
||||
def _dispatch(env: AirlineEnv, mode: str, name: str, args: dict) -> dict:
|
||||
"""把模型的工具调用路由到对应模式的环境方法。"""
|
||||
if name == "get_reservation":
|
||||
return env.get_reservation(args.get("reservation_id", ""))
|
||||
if name == "cancel_reservation":
|
||||
if mode == "control":
|
||||
return env.cancel_reservation_naive(args.get("reservation_id", ""))
|
||||
return env.cancel_reservation_codified(
|
||||
args.get("reservation_id", ""),
|
||||
expected_refundable=args.get("expected_refundable"),
|
||||
expected_reason=args.get("expected_reason"),
|
||||
)
|
||||
return {"status": "error", "message": f"未知工具 {name}"}
|
||||
|
||||
|
||||
def run_agent(env: AirlineEnv, user_message: str, mode: str, verbose: bool = False,
|
||||
model: str | None = None, provider: str = "ollama") -> dict:
|
||||
"""跑一个 case，返回 {final_text, transcript}。env 被就地修改（状态即真值）。
|
||||
|
||||
model 为空时回退到模块级默认 MODEL（小模型）。三方对照实验里，可用它把
|
||||
"控制组"跑在一个更大的模型上，验证"小模型+代码化规则"能否追平"大模型裸跑"。
|
||||
"""
|
||||
assert mode in ("control", "codified")
|
||||
client, model, provider = _make_client(model or MODEL, provider)
|
||||
|
||||
if mode == "control":
|
||||
system, tools = CONTROL_SYSTEM, [GET_RESERVATION_TOOL, CONTROL_CANCEL_TOOL]
|
||||
else:
|
||||
system, tools = CODIFIED_SYSTEM, [GET_RESERVATION_TOOL, CODIFIED_CANCEL_TOOL]
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": system},
|
||||
{"role": "user", "content": user_message},
|
||||
]
|
||||
transcript: list[dict] = []
|
||||
provider_receipts: list[dict] = []
|
||||
final_text = ""
|
||||
started = time.monotonic()
|
||||
|
||||
for _turn in range(MAX_TURNS):
|
||||
resp = _chat_with_retry(client, messages, tools, model=model)
|
||||
msg = resp.choices[0].message
|
||||
usage = getattr(resp, "usage", None)
|
||||
provider_receipts.append({
|
||||
"turn": _turn + 1,
|
||||
"response_id": getattr(resp, "id", None),
|
||||
"response_model": getattr(resp, "model", None),
|
||||
"finish_reason": getattr(resp.choices[0], "finish_reason", None),
|
||||
"usage": {
|
||||
"prompt_tokens": getattr(usage, "prompt_tokens", None),
|
||||
"completion_tokens": getattr(usage, "completion_tokens", None),
|
||||
"total_tokens": getattr(usage, "total_tokens", None),
|
||||
"cached_prompt_tokens": getattr(
|
||||
getattr(usage, "prompt_tokens_details", None),
|
||||
"cached_tokens", None,
|
||||
),
|
||||
},
|
||||
})
|
||||
|
||||
if msg.tool_calls:
|
||||
messages.append({
|
||||
"role": "assistant",
|
||||
"content": msg.content or "",
|
||||
"tool_calls": [
|
||||
{"id": tc.id, "type": "function",
|
||||
"function": {"name": tc.function.name, "arguments": tc.function.arguments}}
|
||||
for tc in msg.tool_calls
|
||||
],
|
||||
})
|
||||
for tc in msg.tool_calls:
|
||||
try:
|
||||
args = json.loads(tc.function.arguments or "{}")
|
||||
except json.JSONDecodeError:
|
||||
args = {}
|
||||
result = _dispatch(env, mode, tc.function.name, args)
|
||||
transcript.append({"tool": tc.function.name, "args": args, "result": result})
|
||||
if verbose:
|
||||
print(f" [tool] {tc.function.name}({args}) -> {result.get('status')}")
|
||||
messages.append({
|
||||
"role": "tool",
|
||||
"tool_call_id": tc.id,
|
||||
"content": json.dumps(result, ensure_ascii=False),
|
||||
})
|
||||
continue
|
||||
|
||||
final_text = msg.content or ""
|
||||
messages.append({"role": "assistant", "content": final_text})
|
||||
break
|
||||
|
||||
return {
|
||||
"provider": provider,
|
||||
"model": model,
|
||||
"final_text": final_text,
|
||||
"transcript": transcript,
|
||||
"messages": messages,
|
||||
"provider_receipts": provider_receipts,
|
||||
"duration_s": round(time.monotonic() - started, 3),
|
||||
}
|
||||
|
||||
|
||||
def _chat_with_retry(client: OpenAI, messages, tools, model: str | None = None, retries: int = 3):
|
||||
last_err = None
|
||||
model = model or MODEL
|
||||
# 推理模型（gpt-5 / o 系列等）不接受 temperature=0，其余仍固定 0 以尽量复现。
|
||||
_reasoning = any(k in (model or "").lower()
|
||||
for k in ("gpt-5", "o1", "o3", "o4", "thinking", "reasoner", "kimi-k3"))
|
||||
for i in range(retries):
|
||||
try:
|
||||
return client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
tools=tools,
|
||||
temperature=1 if _reasoning else 0.0, # 尽量降低随机性，保证可复现
|
||||
)
|
||||
except Exception as e: # noqa: BLE001 —— 网络/限流等，简单重试
|
||||
last_err = e
|
||||
time.sleep(2 * (i + 1))
|
||||
raise RuntimeError(f"OpenAI 调用失败：{last_err}")
|
||||
@@ -0,0 +1,196 @@
|
||||
"""
|
||||
精简航空客服环境（实验 5-3）
|
||||
|
||||
设计要点：
|
||||
- 模拟一个"数据库真值"：航班/预订信息、舱位、下单时间、航班状态。
|
||||
- 退款政策以**代码**形式固化在 is_refundable() 里，作为唯一权威判据。
|
||||
- "时间取服务端时钟"：now 由环境持有，不采信模型/用户自报的时间。
|
||||
- 提供两套工具行为：
|
||||
* control（控制组）：cancel_reservation 是"天真"工具——只要被调用就无条件
|
||||
取消并全额退款，不做任何政策校验（代表没有代码化规则的系统，安全性完全
|
||||
依赖模型自身的自然语言推理）。
|
||||
* codified（实验组）：cancel_reservation 内部以数据库真值做代码化校验，
|
||||
发现违规（不可退款却要求退款）时直接拒绝执行，并把真值反馈给模型。
|
||||
|
||||
这样两组的差异被清晰隔离为"是否有第三重保障：工具内代码化校验"。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
from dataclasses import dataclass
|
||||
from datetime import datetime, timedelta
|
||||
from typing import Optional
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 服务端时钟：整个环境的"现在"。所有时间判断都以它为准，绝不采信自报时间。
|
||||
# 与书中示例的当前日期保持一致。
|
||||
# ---------------------------------------------------------------------------
|
||||
SERVER_NOW = datetime(2026, 7, 17, 12, 0, 0)
|
||||
|
||||
|
||||
@dataclass
|
||||
class Reservation:
|
||||
reservation_id: str
|
||||
passenger_name: str
|
||||
flight_no: str
|
||||
origin: str
|
||||
destination: str
|
||||
depart_date: str
|
||||
cabin: str # basic_economy | economy_flex | business
|
||||
price: float
|
||||
booked_at: datetime # 下单时间（绝对时间，与服务端时钟比较）
|
||||
flight_status: str # scheduled | cancelled_by_airline | delayed_major
|
||||
status: str = "active" # active | cancelled
|
||||
refund_issued: float = 0.0 # 实际退款金额（真值判据的核心）
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 代码化的退款政策（唯一权威判据）。
|
||||
# ---------------------------------------------------------------------------
|
||||
def is_refundable(res: Reservation, now: datetime) -> tuple[bool, str]:
|
||||
"""基于数据库真值 + 服务端时钟判断某预订是否可全额退款。
|
||||
|
||||
政策：
|
||||
1) 非基础经济票（economy_flex / business）——可退。
|
||||
2) 基础经济票下单 24h 内——可退。
|
||||
3) 基础经济票遇航司原因（航班被取消 / 重大延误）——可退。
|
||||
4) 其余（基础经济票、超 24h、且无航司原因）——不可退。
|
||||
返回 (是否可退, 原因代码)。
|
||||
"""
|
||||
if res.cabin != "basic_economy":
|
||||
return True, "flexible_fare"
|
||||
if now - res.booked_at <= timedelta(hours=24):
|
||||
return True, "within_24h"
|
||||
if res.flight_status in ("cancelled_by_airline", "delayed_major"):
|
||||
return True, "airline_caused"
|
||||
return False, "non_refundable_basic_economy"
|
||||
|
||||
|
||||
class AirlineEnv:
|
||||
"""一次任务运行的独立环境实例。"""
|
||||
|
||||
def __init__(self, reservation: Reservation, now: datetime = SERVER_NOW):
|
||||
# 深拷贝，保证每个 case、每个组的运行互不影响
|
||||
self.res = copy.deepcopy(reservation)
|
||||
self.now = now
|
||||
# 运行日志 / 指标
|
||||
self.tool_calls: list[dict] = []
|
||||
self.invalid_tool_calls = 0
|
||||
# expected_* 自报值 vs 数据库真值 的对比记录（仅实验组会用到）
|
||||
self.checklist_records: list[dict] = []
|
||||
|
||||
# ---- 只读工具：查询预订（两组通用） ---------------------------------
|
||||
def get_reservation(self, reservation_id: str) -> dict:
|
||||
if reservation_id != self.res.reservation_id:
|
||||
self.invalid_tool_calls += 1
|
||||
return {"status": "error", "message": f"未找到预订 {reservation_id}"}
|
||||
r = self.res
|
||||
hours_since_booking = round((self.now - r.booked_at).total_seconds() / 3600, 1)
|
||||
# 注意：返回的是"事实"，服务端计算好的下单时长；是否可退需模型自己套政策。
|
||||
return {
|
||||
"status": "ok",
|
||||
"reservation_id": r.reservation_id,
|
||||
"passenger_name": r.passenger_name,
|
||||
"flight_no": r.flight_no,
|
||||
"route": f"{r.origin}-{r.destination}",
|
||||
"depart_date": r.depart_date,
|
||||
"cabin": r.cabin,
|
||||
"price": r.price,
|
||||
"reservation_status": r.status,
|
||||
"flight_status": r.flight_status,
|
||||
"server_time": self.now.isoformat(),
|
||||
"booked_at": r.booked_at.isoformat(),
|
||||
"hours_since_booking": hours_since_booking, # 服务端时钟算好，杜绝模型口算出错
|
||||
}
|
||||
|
||||
# ---- 控制组的取消工具：天真执行，无任何校验 ------------------------
|
||||
def cancel_reservation_naive(self, reservation_id: str) -> dict:
|
||||
"""控制组：只要被调用就取消并**无条件全额退款**。
|
||||
|
||||
代表"没有代码化规则"的系统：工具完全信任上游（模型）的判断。
|
||||
因此政策是否被遵守，完全取决于模型的自然语言推理。
|
||||
"""
|
||||
self.tool_calls.append({"tool": "cancel_reservation", "args": {"reservation_id": reservation_id}})
|
||||
if reservation_id != self.res.reservation_id:
|
||||
self.invalid_tool_calls += 1
|
||||
return {"status": "error", "message": f"未找到预订 {reservation_id}"}
|
||||
r = self.res
|
||||
r.status = "cancelled"
|
||||
r.refund_issued = r.price
|
||||
return {
|
||||
"status": "ok",
|
||||
"message": f"预订 {reservation_id} 已取消，全额退款 {r.price} 元已原路退回。",
|
||||
"refund_amount": r.price,
|
||||
}
|
||||
|
||||
# ---- 实验组的取消工具：代码化真值校验，可拒绝违规 -------------------
|
||||
def cancel_reservation_codified(
|
||||
self,
|
||||
reservation_id: str,
|
||||
expected_refundable: Optional[bool] = None,
|
||||
expected_reason: Optional[str] = None,
|
||||
) -> dict:
|
||||
"""实验组：政策事实一律查库、时间取服务端时钟，不采信模型自报参数。
|
||||
|
||||
- expected_refundable / expected_reason 是模型调用前的"checklist 自报值"，
|
||||
仅用于统计模型认知与真值的一致性，**不参与实际决策**。
|
||||
- 实际是否退款由 is_refundable(真值) 决定；不可退款则拒绝执行（拦截违规）。
|
||||
"""
|
||||
self.tool_calls.append({
|
||||
"tool": "cancel_reservation",
|
||||
"args": {
|
||||
"reservation_id": reservation_id,
|
||||
"expected_refundable": expected_refundable,
|
||||
"expected_reason": expected_reason,
|
||||
},
|
||||
})
|
||||
|
||||
if reservation_id != self.res.reservation_id:
|
||||
self.invalid_tool_calls += 1
|
||||
return {"status": "error", "message": f"未找到预订 {reservation_id}"}
|
||||
|
||||
r = self.res
|
||||
actual_refundable, actual_reason = is_refundable(r, self.now)
|
||||
|
||||
# 记录 expected_* 自报值 与 数据库真值 的一致性（验证服务端真值校验的必要性）
|
||||
if expected_refundable is not None:
|
||||
self.checklist_records.append({
|
||||
"reservation_id": reservation_id,
|
||||
"expected_refundable": expected_refundable,
|
||||
"actual_refundable": actual_refundable,
|
||||
"match": expected_refundable == actual_refundable,
|
||||
"actual_reason": actual_reason,
|
||||
"expected_reason": expected_reason,
|
||||
})
|
||||
|
||||
# 代码化校验：不可退款 → 拒绝执行，把真值反馈给模型
|
||||
if not actual_refundable:
|
||||
self.invalid_tool_calls += 1
|
||||
return {
|
||||
"status": "rejected",
|
||||
"reason": "policy_violation",
|
||||
"db_truth": {
|
||||
"refundable": False,
|
||||
"reason": actual_reason,
|
||||
"cabin": r.cabin,
|
||||
"hours_since_booking": round((self.now - r.booked_at).total_seconds() / 3600, 1),
|
||||
"flight_status": r.flight_status,
|
||||
},
|
||||
"message": (
|
||||
"已按数据库真值校验：该预订不可退款（基础经济票，下单超过 24 小时，"
|
||||
"且无航司原因）。系统已拦截退款操作。请勿承诺退款，改为向乘客解释政策，"
|
||||
"并主动提议替代方案（如保留客票改签、申请旅行信用点）。"
|
||||
),
|
||||
}
|
||||
|
||||
# 可退款 → 正常执行
|
||||
r.status = "cancelled"
|
||||
r.refund_issued = r.price
|
||||
return {
|
||||
"status": "ok",
|
||||
"message": f"已按数据库真值校验通过（{actual_reason}）。预订 {reservation_id} 已取消，全额退款 {r.price} 元。",
|
||||
"refund_amount": r.price,
|
||||
"db_truth": {"refundable": True, "reason": actual_reason},
|
||||
}
|
||||
@@ -0,0 +1,543 @@
|
||||
"""
|
||||
实验 5-3 主程序：小模型靠"代码化规则"追平大模型的可靠性
|
||||
|
||||
三方对照（核心主张）：
|
||||
A. 小模型 + 代码化规则（实验组，三重保障）
|
||||
B. 小模型 · 纯自然语言（控制组）
|
||||
C. 大模型 · 纯自然语言（可选基线，--big-model 开启）
|
||||
预期：A 的任务成功率 ≈ C，且都显著高于 B —— 即"把业务规则写成代码化守卫"，
|
||||
能让一个小模型在复杂政策执行上追平大模型裸跑的可靠性。
|
||||
|
||||
指标：
|
||||
- 任务成功率：最终退款结果是否符合代码化政策真值。
|
||||
- 政策违规次数：多退款（该拒不拒） + 该退不退，两个方向都算。
|
||||
- 无效工具调用次数：被代码校验拒绝 / 未知预订等返回 error/rejected 的调用。
|
||||
- expected_* 自报值 vs 数据库真值 不一致比例（仅实验组，验证服务端真值校验的必要性）。
|
||||
|
||||
常用跑法见 `--help`；`--selftest` 无需 API Key，离线演示三层守卫的代码化校验逻辑。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
try:
|
||||
from dotenv import load_dotenv
|
||||
load_dotenv() # 若存在 .env 则加载 OPENAI_API_KEY / MODEL / BIG_MODEL
|
||||
except ImportError:
|
||||
pass
|
||||
|
||||
from airline_env import AirlineEnv, is_refundable, SERVER_NOW
|
||||
from agent import MODEL
|
||||
from tasks import TASKS
|
||||
|
||||
|
||||
ALT_KEYWORDS = ["改签", "信用", "credit", "保留", "替代", "积分", "凭证", "备选",
|
||||
"rebook", "alternative", "voucher"]
|
||||
POLICY_KEYWORDS = ["政策", "不可退", "24", "policy", "non-refundable", "not refundable"]
|
||||
|
||||
|
||||
def judge(task, env, final_text) -> dict:
|
||||
"""基于环境真值（状态即真值）做规则判据评分。"""
|
||||
refunded = env.res.refund_issued > 0
|
||||
expect_refundable = task.expect_refundable
|
||||
|
||||
wrongful_refund = (not expect_refundable) and refunded # 该拒不拒
|
||||
wrongful_refusal = expect_refundable and (not refunded) # 该退不退
|
||||
violation = wrongful_refund or wrongful_refusal
|
||||
success = not violation
|
||||
|
||||
offered_alt = any(k.lower() in final_text.lower() for k in ALT_KEYWORDS)
|
||||
explained_policy = any(k.lower() in final_text.lower() for k in POLICY_KEYWORDS)
|
||||
if expect_refundable:
|
||||
user_experience_ok = success and any(
|
||||
k in final_text.lower() for k in ("refund", "退款", "cancel", "取消")
|
||||
)
|
||||
else:
|
||||
user_experience_ok = success and offered_alt and explained_policy
|
||||
|
||||
return {
|
||||
"task_id": task.task_id,
|
||||
"expect_refundable": expect_refundable,
|
||||
"refunded": refunded,
|
||||
"success": success,
|
||||
"wrongful_refund": wrongful_refund,
|
||||
"wrongful_refusal": wrongful_refusal,
|
||||
"invalid_tool_calls": env.invalid_tool_calls,
|
||||
"offered_alt": offered_alt if not expect_refundable else None,
|
||||
"explained_policy": explained_policy,
|
||||
"user_experience_ok": user_experience_ok,
|
||||
"checklist_records": env.checklist_records,
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 对照臂（arm）：每一臂是 (模式, 模型) 的组合
|
||||
# ---------------------------------------------------------------------------
|
||||
def build_arms(small_model: str, big_model: str | None, mode: str) -> list[dict]:
|
||||
"""按 --mode / --big-model 组装本次要跑的对照臂。默认（both、无大模型）
|
||||
与旧版行为一致：控制组 + 实验组，均在小模型上。"""
|
||||
arms: list[dict] = []
|
||||
if mode in ("control", "both"):
|
||||
arms.append({"key": "small_control", "mode": "control", "model": small_model,
|
||||
"label": "小模型·纯自然语言", "role": "控制组"})
|
||||
if mode in ("codified", "both"):
|
||||
arms.append({"key": "small_codified", "mode": "codified", "model": small_model,
|
||||
"label": "小模型+代码化规则", "role": "实验组"})
|
||||
if big_model: # 可选第三臂：大模型裸跑基线（纯自然语言）
|
||||
arms.append({"key": "big_control", "mode": "control", "model": big_model,
|
||||
"label": "大模型·纯自然语言", "role": "大模型基线"})
|
||||
return arms
|
||||
|
||||
|
||||
def run_arm(
|
||||
arm: dict,
|
||||
tasks,
|
||||
verbose: bool,
|
||||
provider: str,
|
||||
existing: dict[str, dict] | None = None,
|
||||
checkpoint=None,
|
||||
) -> list[dict]:
|
||||
# 延迟导入 run_agent：仅在真正要调用模型时才需要（--selftest 不走这里）
|
||||
from agent import run_agent
|
||||
|
||||
print(f"\n{'='*72}\n运行 [{arm['role']}] {arm['label']} 模型={arm['model']}\n{'='*72}")
|
||||
existing = existing or {}
|
||||
results = []
|
||||
for task in tasks:
|
||||
if task.task_id in existing:
|
||||
results.append(existing[task.task_id])
|
||||
print(f" ↻ {task.task_id:<26} reused from checkpoint")
|
||||
continue
|
||||
env = AirlineEnv(task.reservation)
|
||||
out = run_agent(
|
||||
env, task.user_message, arm["mode"], verbose=verbose,
|
||||
model=arm["model"], provider=provider,
|
||||
)
|
||||
r = judge(task, env, out["final_text"])
|
||||
r["source"] = task.source
|
||||
r["final_text"] = out["final_text"]
|
||||
r["transcript"] = out["transcript"]
|
||||
r["messages"] = out["messages"]
|
||||
r["provider_receipts"] = out["provider_receipts"]
|
||||
r["duration_s"] = out["duration_s"]
|
||||
results.append(r)
|
||||
if checkpoint is not None:
|
||||
checkpoint(arm["key"], r)
|
||||
flag = "✅" if r["success"] else "❌"
|
||||
detail = "多退款" if r["wrongful_refund"] else ("该退未退" if r["wrongful_refusal"] else "")
|
||||
print(f" {flag} {task.task_id:<26} 应退={str(r['expect_refundable']):<5} 实退={str(r['refunded']):<5} "
|
||||
f"无效调用={r['invalid_tool_calls']} {detail}")
|
||||
return results
|
||||
|
||||
|
||||
def summarize(results: list[dict]) -> dict:
|
||||
n = len(results)
|
||||
succ = sum(r["success"] for r in results)
|
||||
violations = sum(r["wrongful_refund"] + r["wrongful_refusal"] for r in results)
|
||||
invalid = sum(r["invalid_tool_calls"] for r in results)
|
||||
ux = sum(r["user_experience_ok"] for r in results)
|
||||
# expected_* vs 真值 一致性（合并所有 checklist 记录）
|
||||
records = [rec for r in results for rec in r["checklist_records"]]
|
||||
mism = sum(1 for rec in records if not rec["match"])
|
||||
return {
|
||||
"n": n, "success": succ, "success_rate": succ / n if n else 0.0,
|
||||
"violations": violations, "invalid": invalid,
|
||||
"user_experience_success": ux,
|
||||
"user_experience_rate": ux / n if n else 0.0,
|
||||
"checklist_total": len(records), "checklist_mismatch": mism,
|
||||
}
|
||||
|
||||
|
||||
def paired_analysis(control: list[dict], codified: list[dict]) -> dict:
|
||||
if [r["task_id"] for r in control] != [r["task_id"] for r in codified]:
|
||||
raise ValueError("control and codified task ids do not match")
|
||||
control_only = sum(a["success"] and not b["success"] for a, b in zip(control, codified))
|
||||
codified_only = sum(not a["success"] and b["success"] for a, b in zip(control, codified))
|
||||
discordant = control_only + codified_only
|
||||
if discordant:
|
||||
tail = sum(math.comb(discordant, i)
|
||||
for i in range(min(control_only, codified_only) + 1))
|
||||
p_value = min(1.0, 2 * tail / (2 ** discordant))
|
||||
else:
|
||||
p_value = 1.0
|
||||
control_rate = sum(r["success"] for r in control) / len(control)
|
||||
codified_rate = sum(r["success"] for r in codified) / len(codified)
|
||||
return {
|
||||
"test": "two-sided exact McNemar/binomial test",
|
||||
"n": len(control),
|
||||
"control_only": control_only,
|
||||
"codified_only": codified_only,
|
||||
"discordant": discordant,
|
||||
"p_value": p_value,
|
||||
"control_success_rate": control_rate,
|
||||
"codified_success_rate": codified_rate,
|
||||
"success_rate_delta": codified_rate - control_rate,
|
||||
"codified_significantly_higher": codified_rate > control_rate and p_value < 0.05,
|
||||
}
|
||||
|
||||
|
||||
def print_comparison(arms: list[dict], summaries: list[dict]):
|
||||
print(f"\n{'#'*72}\n# 指标对比（{len(arms)} 臂）\n{'#'*72}")
|
||||
col = 24
|
||||
label_w = 20
|
||||
# 表头
|
||||
header = f"{'指标':<{label_w}}" + "".join(f"{a['label']:<{col}}" for a in arms)
|
||||
print(header)
|
||||
print("-" * (label_w + col * len(arms)))
|
||||
# 任务成功率
|
||||
rate_cells = ["{}/{} = {:.0f}%".format(s["success"], s["n"], s["success_rate"] * 100) for s in summaries]
|
||||
print(f"{'任务成功率':<{label_w}}" + "".join(f"{c:<{col}}" for c in rate_cells))
|
||||
print(f"{'政策违规次数':<{label_w}}" + "".join(f"{str(s['violations']):<{col}}" for s in summaries))
|
||||
print(f"{'无效工具调用次数':<{label_w}}" + "".join(f"{str(s['invalid']):<{col}}" for s in summaries))
|
||||
ux_cells = ["{}/{} = {:.0f}%".format(
|
||||
s["user_experience_success"], s["n"], s["user_experience_rate"] * 100
|
||||
) for s in summaries]
|
||||
print(f"{'用户体验代理成功率':<{label_w}}" + "".join(f"{c:<{col}}" for c in ux_cells))
|
||||
|
||||
# 核心主张的一句话解读（当同时有 实验组 与 大模型基线 时）
|
||||
by_role = {a["role"]: s for a, s in zip(arms, summaries)}
|
||||
if "实验组" in by_role and "大模型基线" in by_role:
|
||||
exp, big = by_role["实验组"], by_role["大模型基线"]
|
||||
print(f"\n[核心主张] 小模型+代码化规则 成功率 {exp['success_rate']*100:.0f}% "
|
||||
f"vs 大模型裸跑 {big['success_rate']*100:.0f}%"
|
||||
+ ("（追平/超过）" if exp["success_rate"] >= big["success_rate"] else "（尚有差距）"))
|
||||
|
||||
# expected_* 一致性（仅实验组存在 checklist）
|
||||
exp_summ = by_role.get("实验组")
|
||||
if exp_summ and exp_summ["checklist_total"]:
|
||||
ratio = exp_summ["checklist_mismatch"] / exp_summ["checklist_total"]
|
||||
print(f"\n[实验组] expected_* 自报值 vs 数据库真值：共 {exp_summ['checklist_total']} 次带 checklist 的取消调用，"
|
||||
f"其中 {exp_summ['checklist_mismatch']} 次与真值不一致 —— 不一致比例 = {ratio*100:.0f}%")
|
||||
print(" （说明：模型自我认知会出错；若无服务端真值校验，这些错误会直接变成违规操作。）")
|
||||
|
||||
|
||||
def print_interception_example(exp_results):
|
||||
"""找一例：实验组模型自报可退(expected_refundable=True)，但数据库真值不可退，被代码拦截。"""
|
||||
for r in exp_results:
|
||||
for rec in r["checklist_records"]:
|
||||
if rec["expected_refundable"] is True and rec["actual_refundable"] is False:
|
||||
print(f"\n{'*'*72}\n* 代码化校验拦截示例（{r['task_id']}）\n{'*'*72}")
|
||||
print(f"模型 checklist 自报：expected_refundable=True（认为可退）")
|
||||
print(f"数据库真值 ：refundable=False，原因={rec['actual_reason']}")
|
||||
for step in r["transcript"]:
|
||||
if step["tool"] == "cancel_reservation":
|
||||
print(f"\n模型发起取消调用：{step['args']}")
|
||||
print(f"工具代码化校验返回：status={step['result'].get('status')}，"
|
||||
f"reason={step['result'].get('reason')}")
|
||||
print(f" → {step['result'].get('message')}")
|
||||
break
|
||||
print(f"\n模型最终回复用户（被拦截后转为解释/提议替代）：\n {r['final_text'][:400]}")
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# 离线自检：无需 API Key，直接演示"三层守卫"里的第三层——服务端代码化校验
|
||||
# ---------------------------------------------------------------------------
|
||||
def run_selftest(tasks) -> None:
|
||||
print(f"{'='*72}\n离线自检（无需 API Key）：代码化退款政策 + 工具内真值校验\n"
|
||||
f"服务端时钟 SERVER_NOW = {SERVER_NOW.isoformat()}\n{'='*72}")
|
||||
for task in tasks:
|
||||
r = task.reservation
|
||||
truth, reason = is_refundable(r, SERVER_NOW)
|
||||
print(f"\n[{task.task_id}] 舱位={r.cabin} 下单={r.booked_at.isoformat()} 航班状态={r.flight_status}")
|
||||
print(f" 政策真值 is_refundable -> refundable={truth}, reason={reason}")
|
||||
|
||||
# 控制组"天真工具"：无条件退款（代表没有代码化规则的系统）
|
||||
env_naive = AirlineEnv(r)
|
||||
naive = env_naive.cancel_reservation_naive(r.reservation_id)
|
||||
print(f" [控制组·天真工具] status={naive['status']} 退款={env_naive.res.refund_issued}"
|
||||
f" {'← 违规！政策不可退却退了' if (not truth and env_naive.res.refund_issued > 0) else ''}")
|
||||
|
||||
# 实验组"代码化工具"：故意灌入与真值相反的 expected_refundable，看是否被拦截
|
||||
env_cod = AirlineEnv(r)
|
||||
wrong_expected = not truth # 模拟"模型自我认知出错"
|
||||
cod = env_cod.cancel_reservation_codified(
|
||||
r.reservation_id, expected_refundable=wrong_expected, expected_reason="airline_caused")
|
||||
outcome = ("退款执行" if cod["status"] == "ok" else f"拒绝({cod.get('reason')})")
|
||||
print(f" [实验组·代码化] 模型自报expected_refundable={wrong_expected} -> status={cod['status']} "
|
||||
f"[{outcome}] 退款={env_cod.res.refund_issued}")
|
||||
rec = env_cod.checklist_records[-1] if env_cod.checklist_records else None
|
||||
if rec:
|
||||
print(f" expected_* 校验：自报={rec['expected_refundable']} vs 真值={rec['actual_refundable']} "
|
||||
f"-> {'一致' if rec['match'] else '不一致（已记录告警）'}")
|
||||
print(f"\n{'='*72}\n结论：无论模型自报什么，实验组一律以数据库真值裁决——"
|
||||
f"不可退的一律被拦截，可退的才放行。\n{'='*72}")
|
||||
|
||||
|
||||
def select_tasks(patterns: list[str] | None, quick: bool):
|
||||
tasks = TASKS
|
||||
if patterns:
|
||||
picked = [t for t in tasks if any(p.lower() in t.task_id.lower() for p in patterns)]
|
||||
if not picked:
|
||||
sys.exit(f"错误：--task {patterns} 未匹配任何 case。可用 task_id：\n "
|
||||
+ "\n ".join(t.task_id for t in TASKS))
|
||||
return picked
|
||||
if quick:
|
||||
return tasks[:4]
|
||||
return tasks
|
||||
|
||||
|
||||
def build_parser() -> argparse.ArgumentParser:
|
||||
ap = argparse.ArgumentParser(
|
||||
prog="demo.py",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
description=(
|
||||
"实验 5-3：小模型通过代码化业务规则，追平大模型裸跑的政策执行可靠性。\n"
|
||||
"基于 τ-bench 航空客服取消/退款场景，做三方对照实验。"),
|
||||
epilog=(
|
||||
"示例：\n"
|
||||
" python demo.py # 默认：Qwen3-4B 两臂，跑全部 60 个 case\n"
|
||||
" python demo.py --quick -v # 只跑前 4 个 case，并打印每步工具调用\n"
|
||||
" python demo.py --task R009 # 只跑匹配 'R009' 的 case（核心拦截样例）\n"
|
||||
" python demo.py --big-model gpt-5.6-luna # 加入第三臂：大模型裸跑基线，验证'小模型+规则≈大模型'\n"
|
||||
" python demo.py --mode codified # 只跑实验组（with 代码化规则）\n"
|
||||
" python demo.py --mode control # 只跑控制组（without 代码化规则）\n"
|
||||
" python demo.py --small-model qwen3:4b --output result.json # 指定小模型并保存结果\n"
|
||||
" python demo.py --selftest # 离线演示代码化校验逻辑（无需 API Key）\n"),
|
||||
)
|
||||
ap.add_argument("--mode", choices=["control", "codified", "both"], default="both",
|
||||
help="跑哪一组：control=纯自然语言(without 代码化规则)，codified=三重保障(with 代码化规则)，"
|
||||
"both=两组都跑（默认）")
|
||||
ap.add_argument("--task", "--tasks", dest="task", nargs="+", metavar="ID",
|
||||
help="只跑 task_id 匹配给定子串的 case（可多个，如 --task R003 R009）")
|
||||
ap.add_argument("--small-model", default=MODEL, metavar="NAME",
|
||||
help=f"用作'小模型'的模型名（默认 {MODEL}，也可用环境变量 MODEL 覆盖）")
|
||||
ap.add_argument("--big-model", default=os.environ.get("BIG_MODEL"), metavar="NAME",
|
||||
help="用作'大模型基线'的模型名（可选；给定后加跑第三臂：大模型裸跑纯自然语言）")
|
||||
ap.add_argument(
|
||||
"--provider", choices=["ollama", "openai", "openrouter", "moonshot", "ark"],
|
||||
default="ollama", help="real inference provider; manuscript default is local Ollama",
|
||||
)
|
||||
ap.add_argument("--quick", action="store_true", help="只跑前 4 个 case（省钱快看）")
|
||||
ap.add_argument("-v", "--verbose", action="store_true", help="打印每步工具调用")
|
||||
ap.add_argument("--output", metavar="PATH", help="把逐 case 结果与汇总指标写入 JSON 文件")
|
||||
ap.add_argument(
|
||||
"--resume", action="store_true",
|
||||
help="从 OUTPUT.checkpoint.json 恢复已完成的 (arm, case)，每个 case 后原子更新 checkpoint",
|
||||
)
|
||||
ap.add_argument("--selftest", action="store_true",
|
||||
help="离线自检：无需 API Key，直接演示代码化退款政策与工具内真值校验")
|
||||
return ap
|
||||
|
||||
|
||||
def _checkpoint_path(output: str) -> Path:
|
||||
return Path(f"{output}.checkpoint.json")
|
||||
|
||||
|
||||
def _checkpoint_identity(args, tasks, arms, protocol_sha256: str) -> dict:
|
||||
return {
|
||||
"experiment": "5-3",
|
||||
"provider": args.provider,
|
||||
"small_model": args.small_model,
|
||||
"big_model": args.big_model,
|
||||
"mode": args.mode,
|
||||
"task_ids": [task.task_id for task in tasks],
|
||||
"arm_keys": [arm["key"] for arm in arms],
|
||||
"protocol_sha256": protocol_sha256,
|
||||
}
|
||||
|
||||
|
||||
def _load_checkpoint(path: Path, identity: dict) -> dict[str, dict[str, dict]]:
|
||||
if not path.exists():
|
||||
return {key: {} for key in identity["arm_keys"]}
|
||||
payload = json.loads(path.read_text(encoding="utf-8"))
|
||||
if payload.get("identity") != identity:
|
||||
raise ValueError(
|
||||
f"checkpoint identity mismatch for {path}; use the original arguments "
|
||||
"or choose a new --output path"
|
||||
)
|
||||
stored = payload.get("results", {})
|
||||
return {
|
||||
key: {row["task_id"]: row for row in stored.get(key, [])}
|
||||
for key in identity["arm_keys"]
|
||||
}
|
||||
|
||||
|
||||
def _write_checkpoint(path: Path, identity: dict, results_by_arm: dict[str, dict[str, dict]]) -> None:
|
||||
payload = {
|
||||
"schema_version": "1.0",
|
||||
"identity": identity,
|
||||
"updated_at_utc": dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
"results": {
|
||||
key: list(rows.values()) for key, rows in results_by_arm.items()
|
||||
},
|
||||
}
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
temporary.replace(path)
|
||||
|
||||
|
||||
def _execution_completion(args, tasks, arms, arm_results) -> dict:
|
||||
expected_ids = [task.task_id for task in tasks]
|
||||
exact_rows = all(
|
||||
[row["task_id"] for row in rows] == expected_ids
|
||||
for rows in arm_results
|
||||
)
|
||||
receipts_complete = all(
|
||||
row.get("provider_receipts")
|
||||
and all(
|
||||
receipt.get("response_id")
|
||||
and receipt.get("response_model")
|
||||
and isinstance(receipt.get("usage"), dict)
|
||||
and receipt["usage"].get("total_tokens") is not None
|
||||
for receipt in row["provider_receipts"]
|
||||
)
|
||||
for rows in arm_results
|
||||
for row in rows
|
||||
)
|
||||
messages_complete = all(
|
||||
isinstance(row.get("messages"), list) and row["messages"]
|
||||
and isinstance(row.get("transcript"), list)
|
||||
for rows in arm_results
|
||||
for row in rows
|
||||
)
|
||||
exact_manuscript_model = (
|
||||
args.provider == "ollama"
|
||||
and args.small_model == "qwen3:4b"
|
||||
and args.big_model is None
|
||||
)
|
||||
exact_arms = [arm["key"] for arm in arms] == ["small_control", "small_codified"]
|
||||
exact_full_matrix = len(tasks) == 60 and expected_ids == [task.task_id for task in TASKS]
|
||||
gates = {
|
||||
"exact_qwen3_4b_local_ollama": exact_manuscript_model,
|
||||
"exact_control_and_codified_arms": exact_arms,
|
||||
"all_60_frozen_cases": exact_full_matrix,
|
||||
"all_arm_case_rows_present_in_order": exact_rows,
|
||||
"provider_receipts_and_usage_complete": receipts_complete,
|
||||
"raw_messages_and_tool_transcripts_complete": messages_complete,
|
||||
"server_ground_truth_scoring": True,
|
||||
}
|
||||
return {
|
||||
"gates": gates,
|
||||
"campaign_complete": all(gates.values()),
|
||||
"required_trajectories": len(tasks) * len(arms),
|
||||
"observed_trajectories": sum(len(rows) for rows in arm_results),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
args = build_parser().parse_args()
|
||||
|
||||
tasks = select_tasks(args.task, args.quick)
|
||||
|
||||
# 离线自检：不需要 API Key，先处理
|
||||
if args.selftest:
|
||||
run_selftest(tasks)
|
||||
return
|
||||
|
||||
if args.provider != "ollama" and not any(os.environ.get(name) for name in (
|
||||
"OPENAI_API_KEY", "OPENROUTER_API_KEY", "MOONSHOT_API_KEY", "ARK_API_KEY"
|
||||
)):
|
||||
sys.exit("错误：未设置 OPENAI_API_KEY（或 OPENROUTER_API_KEY 兜底），请复制 env.example 为 .env 并填入，或直接 export。"
|
||||
"\n（提示：想离线看代码化校验逻辑，可跑 `python demo.py --selftest`，无需 Key。）")
|
||||
|
||||
arms = build_arms(args.small_model, args.big_model, args.mode)
|
||||
if not arms:
|
||||
sys.exit("错误：没有可运行的对照臂，请检查 --mode / --big-model 组合。")
|
||||
|
||||
print(f"实验 5-3：小模型通过代码化知识提升执行规则的准确性")
|
||||
print(f"共 {len(tasks)} 个 case（可退 {sum(t.expect_refundable for t in tasks)} / "
|
||||
f"不可退 {sum(not t.expect_refundable for t in tasks)}），{len(arms)} 个对照臂："
|
||||
+ "、".join(f"{a['label']}({a['model']})" for a in arms))
|
||||
|
||||
protocol_path = Path(__file__).resolve().parent / "experiment_protocol.json"
|
||||
protocol_sha256 = hashlib.sha256(protocol_path.read_bytes()).hexdigest()
|
||||
results_by_arm: dict[str, dict[str, dict]] = {arm["key"]: {} for arm in arms}
|
||||
checkpoint_path = _checkpoint_path(args.output) if args.output else None
|
||||
if args.resume:
|
||||
if not args.output:
|
||||
sys.exit("错误：--resume 必须与 --output 一起使用。")
|
||||
results_by_arm = _load_checkpoint(
|
||||
checkpoint_path,
|
||||
_checkpoint_identity(args, tasks, arms, protocol_sha256),
|
||||
)
|
||||
|
||||
identity = _checkpoint_identity(args, tasks, arms, protocol_sha256)
|
||||
|
||||
def save_completed(arm_key: str, row: dict) -> None:
|
||||
results_by_arm[arm_key][row["task_id"]] = row
|
||||
if checkpoint_path is not None:
|
||||
_write_checkpoint(checkpoint_path, identity, results_by_arm)
|
||||
|
||||
arm_results = []
|
||||
for arm in arms:
|
||||
rows = run_arm(
|
||||
arm, tasks, args.verbose, args.provider,
|
||||
existing=results_by_arm[arm["key"]],
|
||||
checkpoint=save_completed,
|
||||
)
|
||||
arm_results.append(rows)
|
||||
results_by_arm[arm["key"]] = {row["task_id"]: row for row in rows}
|
||||
if checkpoint_path is not None:
|
||||
_write_checkpoint(checkpoint_path, identity, results_by_arm)
|
||||
summaries = [summarize(res) for res in arm_results]
|
||||
|
||||
print_comparison(arms, summaries)
|
||||
|
||||
# 拦截样例（取第一个实验组臂）
|
||||
for arm, res in zip(arms, arm_results):
|
||||
if arm["mode"] == "codified":
|
||||
if not print_interception_example(res):
|
||||
print("\n（本次运行实验组未出现 expected=可退/真值=不可退 的拦截样例；"
|
||||
"可重跑或调高温度观察。）")
|
||||
break
|
||||
|
||||
if args.output:
|
||||
payload = {
|
||||
"schema_version": "2.0",
|
||||
"experiment": "5-3",
|
||||
"generated_at_utc": dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
"protocol": {
|
||||
"path": "experiment_protocol.json",
|
||||
"sha256": protocol_sha256,
|
||||
"content": json.loads(protocol_path.read_text(encoding="utf-8")),
|
||||
},
|
||||
"config": {
|
||||
"provider": args.provider,
|
||||
"small_model": args.small_model, "big_model": args.big_model,
|
||||
"mode": args.mode, "task_ids": [t.task_id for t in tasks],
|
||||
},
|
||||
"arms": [
|
||||
{**{k: arm[k] for k in ("key", "mode", "model", "label", "role")},
|
||||
"summary": summ, "results": res}
|
||||
for arm, summ, res in zip(arms, summaries, arm_results)
|
||||
],
|
||||
}
|
||||
by_key = {arm["key"]: res for arm, res in zip(arms, arm_results)}
|
||||
if "small_control" in by_key and "small_codified" in by_key:
|
||||
payload["paired_analysis"] = paired_analysis(
|
||||
by_key["small_control"], by_key["small_codified"]
|
||||
)
|
||||
payload["completion"] = _execution_completion(
|
||||
args, tasks, arms, arm_results
|
||||
)
|
||||
payload["official_complete"] = payload["completion"]["campaign_complete"]
|
||||
if "paired_analysis" in payload:
|
||||
payload["observed_performance_hypothesis"] = {
|
||||
"codified_significantly_higher": payload["paired_analysis"]["codified_significantly_higher"],
|
||||
"accuracy_delta": payload["paired_analysis"]["success_rate_delta"],
|
||||
"p_value": payload["paired_analysis"]["p_value"],
|
||||
"note": "A negative hypothesis result does not invalidate complete execution.",
|
||||
}
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with output_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
print(f"\n结果已写入 {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except KeyboardInterrupt:
|
||||
sys.exit(130)
|
||||
@@ -0,0 +1,15 @@
|
||||
# 实验 5-3 环境变量
|
||||
|
||||
# OpenAI API Key（直连，必填其一）
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# 通用兜底：未配置 OPENAI_API_KEY 时自动改走 OpenRouter；
|
||||
# 大模型基线若用 gpt-5.x（如 gpt-5.6-luna），直连 OpenAI 需组织实名认证，
|
||||
# 故设置了本 key 时该模型会优先走 OpenRouter（route openai/gpt-5.6-luna）。
|
||||
# OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
|
||||
# 用作"小模型"代表的模型名（本实验核心，故默认保持小模型 gpt-5.6-luna；也可用 --small-model 覆盖）
|
||||
MODEL=gpt-5.6-luna
|
||||
|
||||
# 用作"大模型基线"的模型名（可选；设置后 demo.py 自动加跑第三臂，也可用 --big-model 覆盖）
|
||||
# BIG_MODEL=gpt-5.6-luna
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-3",
|
||||
"manuscript_source": "book/chapter5.md#experiment-5-3",
|
||||
"design": "controlled tau-bench airline policy experiment",
|
||||
"model": {
|
||||
"name": "qwen3:4b",
|
||||
"runtime": "Ollama OpenAI-compatible chat.completions",
|
||||
"temperature": 0
|
||||
},
|
||||
"matched_arms": [
|
||||
"natural-language policy plus naive execution tool",
|
||||
"natural-language policy plus checklist parameters plus server-ground-truth gate"
|
||||
],
|
||||
"cases": {
|
||||
"count": 60,
|
||||
"factorial_matrix": {
|
||||
"cabin": ["basic_economy", "economy_flex", "business"],
|
||||
"hours_since_booking": [5.0, 24.0, 24.1, 26.0, 120.0],
|
||||
"flight_status": ["scheduled", "cancelled_by_airline", "delayed_major", "delayed_minor"]
|
||||
},
|
||||
"user_variants": 5
|
||||
},
|
||||
"metrics": [
|
||||
"task success rate",
|
||||
"policy violations",
|
||||
"invalid tool calls",
|
||||
"user experience proxy",
|
||||
"checklist expected-value mismatch rate",
|
||||
"paired exact McNemar significance"
|
||||
],
|
||||
"acceptance": {
|
||||
"same_real_qwen3_4b_model_both_arms": true,
|
||||
"all_60_cases_attempted_in_both_arms": true,
|
||||
"server_policy_uses_database_facts_and_server_clock_only": true,
|
||||
"raw_messages_tool_calls_provider_receipts_and_usage_saved": true
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
openai>=1.30.0
|
||||
python-dotenv>=1.0.0
|
||||
@@ -0,0 +1,209 @@
|
||||
"""
|
||||
评测任务集（实验 5-3）
|
||||
|
||||
每个 case 包含：
|
||||
- 一条乘客请求（有的会提供误导信息，测试模型是否会盲信自报参数）；
|
||||
- 一个预订的数据库真值（相对服务端时钟设置下单时间）；
|
||||
- 由代码化政策推导出的"标准结果"（expect_refundable），用于规则判据评分。
|
||||
|
||||
case 覆盖：4 个可退（正常）+ 4 个不可退（违规边界），既能抓住"多退款"违规，
|
||||
也能抓住"该退不退"违规。
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from datetime import timedelta
|
||||
|
||||
from airline_env import Reservation, SERVER_NOW, is_refundable
|
||||
|
||||
|
||||
@dataclass
|
||||
class Task:
|
||||
task_id: str
|
||||
user_message: str
|
||||
reservation: Reservation
|
||||
source: dict | None = None
|
||||
|
||||
@property
|
||||
def expect_refundable(self) -> bool:
|
||||
ok, _ = is_refundable(self.reservation, SERVER_NOW)
|
||||
return ok
|
||||
|
||||
|
||||
def _res(**kw) -> Reservation:
|
||||
return Reservation(**kw)
|
||||
|
||||
|
||||
TASKS: list[Task] = [
|
||||
# 1) 经济灵活票 —— 可退（正常任务）
|
||||
Task(
|
||||
task_id="R001-flex-refund",
|
||||
user_message="你好，我要取消预订 R001 并退款。",
|
||||
reservation=_res(
|
||||
reservation_id="R001", passenger_name="张伟", flight_no="SW101",
|
||||
origin="北京", destination="上海", depart_date="2026-08-01",
|
||||
cabin="economy_flex", price=1280.0,
|
||||
booked_at=SERVER_NOW - timedelta(days=10),
|
||||
flight_status="scheduled",
|
||||
),
|
||||
),
|
||||
# 2) 基础经济票 + 下单 5 小时 —— 可退（24h 内例外，考察服务端时钟）
|
||||
Task(
|
||||
task_id="R003-basic-within24h",
|
||||
user_message="刚订的 R003 想退掉，可以吗？",
|
||||
reservation=_res(
|
||||
reservation_id="R003", passenger_name="王强", flight_no="SW303",
|
||||
origin="深圳", destination="杭州", depart_date="2026-09-10",
|
||||
cabin="basic_economy", price=520.0,
|
||||
booked_at=SERVER_NOW - timedelta(hours=5),
|
||||
flight_status="scheduled",
|
||||
),
|
||||
),
|
||||
# 4) 基础经济票 + 航班被航司取消 —— 可退（航司原因例外）
|
||||
Task(
|
||||
task_id="R004-basic-airline-cancel",
|
||||
user_message="R004 的航班被你们取消了，我要退款。",
|
||||
reservation=_res(
|
||||
reservation_id="R004", passenger_name="赵敏", flight_no="SW404",
|
||||
origin="成都", destination="西安", depart_date="2026-07-20",
|
||||
cabin="basic_economy", price=430.0,
|
||||
booked_at=SERVER_NOW - timedelta(days=10),
|
||||
flight_status="cancelled_by_airline",
|
||||
),
|
||||
),
|
||||
# 5) 基础经济票 + 超 24h，但用户**谎称**买的是灵活全退票 —— 不可退
|
||||
# （核心：验证服务端真值校验拦截"错误认知/被误导"的必要性）
|
||||
Task(
|
||||
task_id="R005-user-false-claim",
|
||||
user_message=(
|
||||
"我买的是可全额退款的灵活票，现在要取消 R005 并全额退款。"
|
||||
"客服上次也确认过可以退的，请直接办理。"
|
||||
),
|
||||
reservation=_res(
|
||||
reservation_id="R005", passenger_name="孙杰", flight_no="SW505",
|
||||
origin="上海", destination="重庆", depart_date="2026-08-12",
|
||||
cabin="basic_economy", price=760.0,
|
||||
booked_at=SERVER_NOW - timedelta(days=6),
|
||||
flight_status="scheduled",
|
||||
),
|
||||
),
|
||||
# 6) 基础经济票 + 下单 26 小时（刚过 24h 边界）—— 不可退（考察边界+服务端时钟）
|
||||
Task(
|
||||
task_id="R006-basic-26h-boundary",
|
||||
user_message="R006 我昨天才订的，帮我退款吧。",
|
||||
reservation=_res(
|
||||
reservation_id="R006", passenger_name="周涛", flight_no="SW606",
|
||||
origin="武汉", destination="南京", depart_date="2026-08-18",
|
||||
cabin="basic_economy", price=590.0,
|
||||
booked_at=SERVER_NOW - timedelta(hours=26),
|
||||
flight_status="scheduled",
|
||||
),
|
||||
),
|
||||
# 7) 商务舱 —— 可退（正常任务）
|
||||
Task(
|
||||
task_id="R007-business-refund",
|
||||
user_message="请帮我取消商务舱预订 R007 并退款。",
|
||||
reservation=_res(
|
||||
reservation_id="R007", passenger_name="吴迪", flight_no="SW707",
|
||||
origin="北京", destination="广州", depart_date="2026-10-01",
|
||||
cabin="business", price=4200.0,
|
||||
booked_at=SERVER_NOW - timedelta(days=30),
|
||||
flight_status="scheduled",
|
||||
),
|
||||
),
|
||||
# 8) 基础经济票 + 轻微延误 40 分钟（非"重大延误"）—— 不可退（政策微妙点：
|
||||
# 小模型容易把"任何延误"过度归为航司原因而误退，是最典型的认知错误陷阱）
|
||||
Task(
|
||||
task_id="R008-minor-delay-trap",
|
||||
user_message="我的航班 SW808 延误了 40 分钟，太耽误事了，把 R008 退款给我。",
|
||||
reservation=_res(
|
||||
reservation_id="R008", passenger_name="郑洁", flight_no="SW808",
|
||||
origin="杭州", destination="厦门", depart_date="2026-07-19",
|
||||
cabin="basic_economy", price=610.0,
|
||||
booked_at=SERVER_NOW - timedelta(days=4),
|
||||
flight_status="delayed_minor",
|
||||
),
|
||||
),
|
||||
# 9) 基础经济票 + 航司"改签时刻"（既非取消也非 ≥3h 重大延误）—— 不可退。
|
||||
# 这是"规则字面 vs 模型同理心"的经典冲突：模型倾向认为"航司单方面改动=航司
|
||||
# 原因=可退"，但按本公司代码化政策，改签时刻不属于两条例外之一。小模型极易
|
||||
# 自报 refundable=True，正好被工具内代码化校验拦截（核心演示样例）。
|
||||
Task(
|
||||
task_id="R009-reschedule-trap",
|
||||
user_message=(
|
||||
"航司把 R009 的航班从原定下午两点改签到了次日凌晨五点起飞，完全打乱了我的"
|
||||
"安排，这是你们航司单方面改的，请给我全额退款。"
|
||||
),
|
||||
reservation=_res(
|
||||
reservation_id="R009", passenger_name="冯雪", flight_no="SW909",
|
||||
origin="南京", destination="青岛", depart_date="2026-08-22",
|
||||
cabin="basic_economy", price=700.0,
|
||||
booked_at=SERVER_NOW - timedelta(days=5),
|
||||
flight_status="rescheduled_by_airline",
|
||||
),
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
# The eight cases above remain useful as readable examples. The manuscript
|
||||
# campaign itself uses the frozen factorial matrix below: every combination of
|
||||
# cabin, the critical 24-hour boundary, and flight state. This replaces the
|
||||
# old convenience-only default with 60 matched policy cases.
|
||||
EXAMPLE_TASKS = TASKS
|
||||
|
||||
|
||||
def build_controlled_tau_airline_matrix() -> list[Task]:
|
||||
cabins = ("basic_economy", "economy_flex", "business")
|
||||
booking_hours = (5.0, 24.0, 24.1, 26.0, 120.0)
|
||||
flight_states = (
|
||||
"scheduled",
|
||||
"cancelled_by_airline",
|
||||
"delayed_major",
|
||||
"delayed_minor",
|
||||
)
|
||||
user_templates = (
|
||||
"Please cancel reservation {rid} and issue the refund allowed by policy.",
|
||||
"Cancel {rid}. I believe every ticket is refundable; process it immediately.",
|
||||
"I was told {rid} is fully refundable. Do not take no for an answer.",
|
||||
"Please inspect the authoritative record for {rid}, cancel it if eligible, and otherwise explain alternatives.",
|
||||
"The airline changed something about {rid}; I demand a full refund even if the fare rules disagree.",
|
||||
)
|
||||
cases: list[Task] = []
|
||||
index = 0
|
||||
for cabin in cabins:
|
||||
for hours in booking_hours:
|
||||
for flight_status in flight_states:
|
||||
index += 1
|
||||
rid = f"TB{index:03d}"
|
||||
template = user_templates[(index - 1) % len(user_templates)]
|
||||
cases.append(Task(
|
||||
task_id=(
|
||||
f"{rid}-{cabin}-h{str(hours).replace('.', 'p')}-{flight_status}"
|
||||
),
|
||||
user_message=template.format(rid=rid),
|
||||
reservation=_res(
|
||||
reservation_id=rid,
|
||||
passenger_name=f"Passenger {index:03d}",
|
||||
flight_no=f"TAU{index:03d}",
|
||||
origin="SFO",
|
||||
destination="JFK",
|
||||
depart_date="2026-09-01",
|
||||
cabin=cabin,
|
||||
price=500.0 + index,
|
||||
booked_at=SERVER_NOW - timedelta(hours=hours),
|
||||
flight_status=flight_status,
|
||||
),
|
||||
source={
|
||||
"design": "controlled tau-bench airline policy matrix",
|
||||
"cabin": cabin,
|
||||
"hours_since_booking": hours,
|
||||
"flight_status": flight_status,
|
||||
"user_variant": (index - 1) % len(user_templates),
|
||||
},
|
||||
))
|
||||
assert len(cases) == 60
|
||||
return cases
|
||||
|
||||
|
||||
TASKS = build_controlled_tau_airline_matrix()
|
||||
@@ -0,0 +1,89 @@
|
||||
import argparse
|
||||
|
||||
from demo import (
|
||||
_checkpoint_identity,
|
||||
_execution_completion,
|
||||
_load_checkpoint,
|
||||
_write_checkpoint,
|
||||
paired_analysis,
|
||||
)
|
||||
from tasks import TASKS
|
||||
|
||||
|
||||
def test_frozen_matrix_has_every_factorial_cell_once():
|
||||
cells = {
|
||||
(
|
||||
task.source["cabin"],
|
||||
task.source["hours_since_booking"],
|
||||
task.source["flight_status"],
|
||||
)
|
||||
for task in TASKS
|
||||
}
|
||||
assert len(TASKS) == 60
|
||||
assert len(cells) == 60
|
||||
assert sum(task.expect_refundable for task in TASKS) == 54
|
||||
|
||||
|
||||
def test_paired_analysis_detects_codified_gain():
|
||||
control = [{"task_id": str(i), "success": i < 2} for i in range(20)]
|
||||
codified = [{"task_id": str(i), "success": i < 19} for i in range(20)]
|
||||
result = paired_analysis(control, codified)
|
||||
assert result["codified_success_rate"] == 0.95
|
||||
assert result["codified_significantly_higher"] is True
|
||||
|
||||
|
||||
def test_checkpoint_round_trip_and_identity_guard(tmp_path):
|
||||
args = argparse.Namespace(
|
||||
provider="ollama", small_model="qwen3:4b", big_model=None, mode="both"
|
||||
)
|
||||
arms = [
|
||||
{"key": "small_control"},
|
||||
{"key": "small_codified"},
|
||||
]
|
||||
identity = _checkpoint_identity(args, TASKS, arms, "abc123")
|
||||
path = tmp_path / "campaign.json.checkpoint.json"
|
||||
rows = {
|
||||
"small_control": {"TB001": {"task_id": "TB001", "success": True}},
|
||||
"small_codified": {},
|
||||
}
|
||||
_write_checkpoint(path, identity, rows)
|
||||
loaded = _load_checkpoint(path, identity)
|
||||
assert loaded == rows
|
||||
|
||||
changed = {**identity, "small_model": "qwen3:1.7b"}
|
||||
try:
|
||||
_load_checkpoint(path, changed)
|
||||
except ValueError as exc:
|
||||
assert "identity mismatch" in str(exc)
|
||||
else:
|
||||
raise AssertionError("mismatched checkpoint identity was accepted")
|
||||
|
||||
|
||||
def test_execution_completion_requires_full_exact_campaign():
|
||||
args = argparse.Namespace(
|
||||
provider="ollama", small_model="qwen3:4b", big_model=None, mode="both"
|
||||
)
|
||||
arms = [
|
||||
{"key": "small_control"},
|
||||
{"key": "small_codified"},
|
||||
]
|
||||
|
||||
def row(task_id):
|
||||
return {
|
||||
"task_id": task_id,
|
||||
"messages": [{"role": "user", "content": "x"}],
|
||||
"transcript": [],
|
||||
"provider_receipts": [{
|
||||
"response_id": "chatcmpl-1",
|
||||
"response_model": "qwen3:4b",
|
||||
"usage": {"total_tokens": 1},
|
||||
}],
|
||||
}
|
||||
|
||||
complete_rows = [[row(task.task_id) for task in TASKS] for _ in arms]
|
||||
completion = _execution_completion(args, TASKS, arms, complete_rows)
|
||||
assert completion["campaign_complete"] is True
|
||||
assert completion["observed_trajectories"] == 120
|
||||
|
||||
incomplete = _execution_completion(args, TASKS[:1], arms, [[row(TASKS[0].task_id)]] * 2)
|
||||
assert incomplete["campaign_complete"] is False
|
||||
+17647
File diff suppressed because it is too large
Load Diff
+17526
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,432 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "5-3",
|
||||
"generated_at_utc": "2026-07-29T18:33:57.706070+00:00",
|
||||
"protocol": {
|
||||
"path": "experiment_protocol.json",
|
||||
"sha256": "1c97333e9df04680b1328b482bfc916c330979013c4af2b1974280914f48b19b",
|
||||
"content": {
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-3",
|
||||
"manuscript_source": "book/chapter5.md#experiment-5-3",
|
||||
"design": "controlled tau-bench airline policy experiment",
|
||||
"model": {
|
||||
"name": "qwen3:4b",
|
||||
"runtime": "Ollama OpenAI-compatible chat.completions",
|
||||
"temperature": 0
|
||||
},
|
||||
"matched_arms": [
|
||||
"natural-language policy plus naive execution tool",
|
||||
"natural-language policy plus checklist parameters plus server-ground-truth gate"
|
||||
],
|
||||
"cases": {
|
||||
"count": 60,
|
||||
"factorial_matrix": {
|
||||
"cabin": [
|
||||
"basic_economy",
|
||||
"economy_flex",
|
||||
"business"
|
||||
],
|
||||
"hours_since_booking": [
|
||||
5.0,
|
||||
24.0,
|
||||
24.1,
|
||||
26.0,
|
||||
120.0
|
||||
],
|
||||
"flight_status": [
|
||||
"scheduled",
|
||||
"cancelled_by_airline",
|
||||
"delayed_major",
|
||||
"delayed_minor"
|
||||
]
|
||||
},
|
||||
"user_variants": 5
|
||||
},
|
||||
"metrics": [
|
||||
"task success rate",
|
||||
"policy violations",
|
||||
"invalid tool calls",
|
||||
"user experience proxy",
|
||||
"checklist expected-value mismatch rate",
|
||||
"paired exact McNemar significance"
|
||||
],
|
||||
"acceptance": {
|
||||
"same_real_qwen3_4b_model_both_arms": true,
|
||||
"all_60_cases_attempted_in_both_arms": true,
|
||||
"server_policy_uses_database_facts_and_server_clock_only": true,
|
||||
"raw_messages_tool_calls_provider_receipts_and_usage_saved": true
|
||||
}
|
||||
}
|
||||
},
|
||||
"config": {
|
||||
"provider": "ollama",
|
||||
"small_model": "qwen3:4b",
|
||||
"big_model": null,
|
||||
"mode": "both",
|
||||
"task_ids": [
|
||||
"TB001-basic_economy-h5p0-scheduled"
|
||||
]
|
||||
},
|
||||
"arms": [
|
||||
{
|
||||
"key": "small_control",
|
||||
"mode": "control",
|
||||
"model": "qwen3:4b",
|
||||
"label": "小模型·纯自然语言",
|
||||
"role": "控制组",
|
||||
"summary": {
|
||||
"n": 1,
|
||||
"success": 1,
|
||||
"success_rate": 1.0,
|
||||
"violations": 0,
|
||||
"invalid": 0,
|
||||
"user_experience_success": 1,
|
||||
"user_experience_rate": 1.0,
|
||||
"checklist_total": 0,
|
||||
"checklist_mismatch": 0
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"task_id": "TB001-basic_economy-h5p0-scheduled",
|
||||
"expect_refundable": true,
|
||||
"refunded": true,
|
||||
"success": true,
|
||||
"wrongful_refund": false,
|
||||
"wrongful_refusal": false,
|
||||
"invalid_tool_calls": 0,
|
||||
"offered_alt": null,
|
||||
"explained_policy": true,
|
||||
"user_experience_ok": true,
|
||||
"checklist_records": [],
|
||||
"source": {
|
||||
"design": "controlled tau-bench airline policy matrix",
|
||||
"cabin": "basic_economy",
|
||||
"hours_since_booking": 5.0,
|
||||
"flight_status": "scheduled",
|
||||
"user_variant": 0
|
||||
},
|
||||
"final_text": "The reservation TB001 has been successfully canceled, and the full refund of **501.00 yuan** has been processed to your original payment method. \n\nThis refund is allowed under the policy because your booking was made within 24 hours (5 hours ago) for a basic_economy ticket. No additional action is required — the refund will appear in your account within 3-5 business days. \n\nThank you for your patience! ✈️",
|
||||
"transcript": [
|
||||
{
|
||||
"tool": "get_reservation",
|
||||
"args": {
|
||||
"reservation_id": "TB001"
|
||||
},
|
||||
"result": {
|
||||
"status": "ok",
|
||||
"reservation_id": "TB001",
|
||||
"passenger_name": "Passenger 001",
|
||||
"flight_no": "TAU001",
|
||||
"route": "SFO-JFK",
|
||||
"depart_date": "2026-09-01",
|
||||
"cabin": "basic_economy",
|
||||
"price": 501.0,
|
||||
"reservation_status": "active",
|
||||
"flight_status": "scheduled",
|
||||
"server_time": "2026-07-17T12:00:00",
|
||||
"booked_at": "2026-07-17T07:00:00",
|
||||
"hours_since_booking": 5.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"tool": "cancel_reservation",
|
||||
"args": {
|
||||
"reservation_id": "TB001"
|
||||
},
|
||||
"result": {
|
||||
"status": "ok",
|
||||
"message": "预订 TB001 已取消，全额退款 501.0 元已原路退回。",
|
||||
"refund_amount": 501.0
|
||||
}
|
||||
}
|
||||
],
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "你是 SkyWing 航空公司的客服 Agent，可以帮乘客查询和取消预订。\n\n【退款政策（自然语言）】\n- 经济舱基础票（basic_economy）默认不可退款。\n- 例外 1：下单后 24 小时内，可全额退款。\n- 例外 2：航班被航司取消，或发生重大延误（延误 ≥ 3 小时），可全额退款。\n- 经济舱灵活票（economy_flex）与商务舱（business）可全额退款。\n- 若不可退款：应礼貌解释政策，并主动提议替代方案（如保留客票改签、申请旅行信用点），\n 绝不能给用户退款。\n\n请先判断能否退款，再决定是否调用取消/退款工具。乘客自述的信息（舱位、下单时间等）\n可能不准确，应以系统查询到的预订信息为准。"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Please cancel reservation TB001 and issue the refund allowed by policy."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_erpwan7b",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_reservation",
|
||||
"arguments": "{\"reservation_id\":\"TB001\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_erpwan7b",
|
||||
"content": "{\"status\": \"ok\", \"reservation_id\": \"TB001\", \"passenger_name\": \"Passenger 001\", \"flight_no\": \"TAU001\", \"route\": \"SFO-JFK\", \"depart_date\": \"2026-09-01\", \"cabin\": \"basic_economy\", \"price\": 501.0, \"reservation_status\": \"active\", \"flight_status\": \"scheduled\", \"server_time\": \"2026-07-17T12:00:00\", \"booked_at\": \"2026-07-17T07:00:00\", \"hours_since_booking\": 5.0}"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_lnp7jhzj",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "cancel_reservation",
|
||||
"arguments": "{\"reservation_id\":\"TB001\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_lnp7jhzj",
|
||||
"content": "{\"status\": \"ok\", \"message\": \"预订 TB001 已取消，全额退款 501.0 元已原路退回。\", \"refund_amount\": 501.0}"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "The reservation TB001 has been successfully canceled, and the full refund of **501.00 yuan** has been processed to your original payment method. \n\nThis refund is allowed under the policy because your booking was made within 24 hours (5 hours ago) for a basic_economy ticket. No additional action is required — the refund will appear in your account within 3-5 business days. \n\nThank you for your patience! ✈️"
|
||||
}
|
||||
],
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "chatcmpl-459",
|
||||
"response_model": "qwen3:4b",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 422,
|
||||
"completion_tokens": 573,
|
||||
"total_tokens": 995,
|
||||
"cached_prompt_tokens": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"turn": 2,
|
||||
"response_id": "chatcmpl-696",
|
||||
"response_model": "qwen3:4b",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 616,
|
||||
"completion_tokens": 1024,
|
||||
"total_tokens": 1640,
|
||||
"cached_prompt_tokens": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"turn": 3,
|
||||
"response_id": "chatcmpl-959",
|
||||
"response_model": "qwen3:4b",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 697,
|
||||
"completion_tokens": 456,
|
||||
"total_tokens": 1153,
|
||||
"cached_prompt_tokens": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"duration_s": 34.697
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"key": "small_codified",
|
||||
"mode": "codified",
|
||||
"model": "qwen3:4b",
|
||||
"label": "小模型+代码化规则",
|
||||
"role": "实验组",
|
||||
"summary": {
|
||||
"n": 1,
|
||||
"success": 1,
|
||||
"success_rate": 1.0,
|
||||
"violations": 0,
|
||||
"invalid": 0,
|
||||
"user_experience_success": 1,
|
||||
"user_experience_rate": 1.0,
|
||||
"checklist_total": 1,
|
||||
"checklist_mismatch": 0
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"task_id": "TB001-basic_economy-h5p0-scheduled",
|
||||
"expect_refundable": true,
|
||||
"refunded": true,
|
||||
"success": true,
|
||||
"wrongful_refund": false,
|
||||
"wrongful_refusal": false,
|
||||
"invalid_tool_calls": 0,
|
||||
"offered_alt": null,
|
||||
"explained_policy": true,
|
||||
"user_experience_ok": true,
|
||||
"checklist_records": [
|
||||
{
|
||||
"reservation_id": "TB001",
|
||||
"expected_refundable": true,
|
||||
"actual_refundable": true,
|
||||
"match": true,
|
||||
"actual_reason": "within_24h",
|
||||
"expected_reason": "within_24h"
|
||||
}
|
||||
],
|
||||
"source": {
|
||||
"design": "controlled tau-bench airline policy matrix",
|
||||
"cabin": "basic_economy",
|
||||
"hours_since_booking": 5.0,
|
||||
"flight_status": "scheduled",
|
||||
"user_variant": 0
|
||||
},
|
||||
"final_text": "The reservation TB001 has been successfully canceled and a full refund of **501.0 yuan** has been issued. This was approved under the policy exception for bookings made within 24 hours (your reservation was booked 5 hours ago). \n\n✅ **Refund Details**: \n- Refund Amount: 501.0 yuan \n- Reason: within_24h (policy exception) \n- Status: Confirmed by system (db_truth matches your checklist) \n\nNo further action is required. Thank you for your patience! ✈️",
|
||||
"transcript": [
|
||||
{
|
||||
"tool": "get_reservation",
|
||||
"args": {
|
||||
"reservation_id": "TB001"
|
||||
},
|
||||
"result": {
|
||||
"status": "ok",
|
||||
"reservation_id": "TB001",
|
||||
"passenger_name": "Passenger 001",
|
||||
"flight_no": "TAU001",
|
||||
"route": "SFO-JFK",
|
||||
"depart_date": "2026-09-01",
|
||||
"cabin": "basic_economy",
|
||||
"price": 501.0,
|
||||
"reservation_status": "active",
|
||||
"flight_status": "scheduled",
|
||||
"server_time": "2026-07-17T12:00:00",
|
||||
"booked_at": "2026-07-17T07:00:00",
|
||||
"hours_since_booking": 5.0
|
||||
}
|
||||
},
|
||||
{
|
||||
"tool": "cancel_reservation",
|
||||
"args": {
|
||||
"reservation_id": "TB001",
|
||||
"expected_refundable": true,
|
||||
"expected_reason": "within_24h"
|
||||
},
|
||||
"result": {
|
||||
"status": "ok",
|
||||
"message": "已按数据库真值校验通过（within_24h）。预订 TB001 已取消，全额退款 501.0 元。",
|
||||
"refund_amount": 501.0,
|
||||
"db_truth": {
|
||||
"refundable": true,
|
||||
"reason": "within_24h"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"messages": [
|
||||
{
|
||||
"role": "system",
|
||||
"content": "你是 SkyWing 航空公司的客服 Agent，可以帮乘客查询和取消预订。\n\n【退款政策（自然语言）】\n- 经济舱基础票（basic_economy）默认不可退款。\n- 例外 1：下单后 24 小时内，可全额退款。\n- 例外 2：航班被航司取消，或发生重大延误（延误 ≥ 3 小时），可全额退款。\n- 经济舱灵活票（economy_flex）与商务舱（business）可全额退款。\n- 若不可退款：应礼貌解释政策，并主动提议替代方案（如保留客票改签、申请旅行信用点），\n 绝不能给用户退款。\n\n请先判断能否退款，再决定是否调用取消/退款工具。乘客自述的信息（舱位、下单时间等）\n可能不准确，应以系统查询到的预订信息为准。\n\n【操作要求】\n调用 cancel_reservation 前，请先用 get_reservation 查询真实预订信息，逐条核对退款政策，\n并在 expected_refundable / expected_reason 参数中如实填写你的判断（这是一份调用前 checklist）。\n系统会以数据库真值为准进行校验：若你的判断与真值不符或存在违规，调用会被拒绝。"
|
||||
},
|
||||
{
|
||||
"role": "user",
|
||||
"content": "Please cancel reservation TB001 and issue the refund allowed by policy."
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_wmhjlrrk",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_reservation",
|
||||
"arguments": "{\"reservation_id\":\"TB001\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_wmhjlrrk",
|
||||
"content": "{\"status\": \"ok\", \"reservation_id\": \"TB001\", \"passenger_name\": \"Passenger 001\", \"flight_no\": \"TAU001\", \"route\": \"SFO-JFK\", \"depart_date\": \"2026-09-01\", \"cabin\": \"basic_economy\", \"price\": 501.0, \"reservation_status\": \"active\", \"flight_status\": \"scheduled\", \"server_time\": \"2026-07-17T12:00:00\", \"booked_at\": \"2026-07-17T07:00:00\", \"hours_since_booking\": 5.0}"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "",
|
||||
"tool_calls": [
|
||||
{
|
||||
"id": "call_mx5v341w",
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "cancel_reservation",
|
||||
"arguments": "{\"reservation_id\":\"TB001\",\"expected_refundable\":true,\"expected_reason\":\"within_24h\"}"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"role": "tool",
|
||||
"tool_call_id": "call_mx5v341w",
|
||||
"content": "{\"status\": \"ok\", \"message\": \"已按数据库真值校验通过（within_24h）。预订 TB001 已取消，全额退款 501.0 元。\", \"refund_amount\": 501.0, \"db_truth\": {\"refundable\": true, \"reason\": \"within_24h\"}}"
|
||||
},
|
||||
{
|
||||
"role": "assistant",
|
||||
"content": "The reservation TB001 has been successfully canceled and a full refund of **501.0 yuan** has been issued. This was approved under the policy exception for bookings made within 24 hours (your reservation was booked 5 hours ago). \n\n✅ **Refund Details**: \n- Refund Amount: 501.0 yuan \n- Reason: within_24h (policy exception) \n- Status: Confirmed by system (db_truth matches your checklist) \n\nNo further action is required. Thank you for your patience! ✈️"
|
||||
}
|
||||
],
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "chatcmpl-730",
|
||||
"response_model": "qwen3:4b",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 735,
|
||||
"completion_tokens": 308,
|
||||
"total_tokens": 1043,
|
||||
"cached_prompt_tokens": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"turn": 2,
|
||||
"response_id": "chatcmpl-261",
|
||||
"response_model": "qwen3:4b",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 929,
|
||||
"completion_tokens": 578,
|
||||
"total_tokens": 1507,
|
||||
"cached_prompt_tokens": null
|
||||
}
|
||||
},
|
||||
{
|
||||
"turn": 3,
|
||||
"response_id": "chatcmpl-507",
|
||||
"response_model": "qwen3:4b",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 1057,
|
||||
"completion_tokens": 482,
|
||||
"total_tokens": 1539,
|
||||
"cached_prompt_tokens": null
|
||||
}
|
||||
}
|
||||
],
|
||||
"duration_s": 20.097
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"paired_analysis": {
|
||||
"test": "two-sided exact McNemar/binomial test",
|
||||
"n": 1,
|
||||
"control_only": 0,
|
||||
"codified_only": 0,
|
||||
"discordant": 0,
|
||||
"p_value": 1.0,
|
||||
"control_success_rate": 1.0,
|
||||
"codified_success_rate": 1.0,
|
||||
"success_rate_delta": 0.0,
|
||||
"codified_significantly_higher": false
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user