# Prompt Engineering Ablation (τ-bench) / 提示工程消融实验

> Companion material for *AI Agents in Depth*, Chapter 2 — **Experiment 2-4 ★★: Ablation study in prompt engineering**.  
> 配套《深入理解 AI Agent》第 2 章 **实验 2-4 ★★：提示工程的消融实验**。

← [Chapter 2 index / 返回第 2 章目录](../README.md)

---

## English

### Overview

Extends the [τ-bench](https://arxiv.org/abs/2406.12045) framework with three ablation axes to show why **prompt engineering treats an Agent like a smart new hire**. Experiments quantify how tone, instruction organization, and tool descriptions affect task success.

### Ablation options

#### 1. Tone style

- **default** — professional baseline  
- **trump** — exaggerated, repetitive, confident phrasing  
- **casual** — emoji/slang, informal  

**Rationale:** Tone affects professionalism and task quality. Over-casual or exaggerated tone can reduce trust, increase misunderstanding, and hurt execution accuracy.

#### 2. Wiki rule randomization

Uses a pre-generated chaotic `wiki.md`:

- Strip section headings/structure  
- Prefix each rule with operation context (e.g. “When booking flights”)  
- Fully shuffle into a flat list  
- Break logical relationships between rules  

**Rationale:** Well-organized instructions are like a training manual. Extreme randomization destroys hierarchy, blurs rule boundaries, and raises misuse/omission risk.

#### 3. Tool description removal

- Empty tool and parameter descriptions  
- Tests the value of explicit documentation  

**Rationale:** Clear tool docs are the “how to use the tools” handbook. Without them the Agent misuses tools more often and completion rates drop.

### Installation

```bash
# From the repository root: use the shared Chapter 2 environment
uv sync --locked --python 3.12 --extra ch2

# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat

# pip fallback when uv is not installed:
# python -m pip install -e ".[ch2]"

cd chapter2/prompt-engineering

# Single-project compatibility path, still supported during migration:
# python -m pip install -r requirements.txt
```

(Older docs may mention `projects/week2/prompt-engineering`; use this repo path.)

### Usage

All entry scripts have Chinese `--help`: `python run_ablation.py --help`, `python analyze_results.py --help`.

#### One-shot full ablation + comparison table (recommended)

`--all` runs baseline + each single-axis ablation + all combined in one process, prints a success-rate table, and writes summary stats to `--output`. The frozen canonical protocol uses official Moonshot Kimi K3 for both the action model and user simulator, six arms, and the same ten τ-bench airline tasks in every arm:

```bash
export OPENAI_API_KEY="$MOONSHOT_API_KEY"
export OPENAI_API_BASE="https://api.moonshot.cn/v1"
python run_ablation.py \
  --all --model kimi-k3 --user-model kimi-k3 \
  --model-provider openai --user-model-provider openai --temperature 1 \
  --env airline --task-ids 0 1 2 3 4 5 6 7 8 9 \
  --num-trials 1 --seed 20260730 --max-agent-steps 30 \
  --max-concurrency 2 \
  --log-dir runs/exp2-4-kimi-k3-YYYYMMDD-v1 \
  --output runs/exp2-4-kimi-k3-YYYYMMDD-v1/comparison.json \
  --no-verbose
```

If a campaign stops, resume into a new evidence directory. The runner imports
only prior task rows with nonempty provider response IDs/usage and no task
error, records the source hash, and never regenerates them:

```bash
# Repeat every frozen option above, change --log-dir/--output to ...-v2, and add:
--resume-from runs/exp2-4-kimi-k3-YYYYMMDD-v1
```

The rejected OpenAI-direct/OpenRouter preflights and any failed tasks remain
evidence; they are not converted into zero-score model outcomes. Campaign
completion requires every arm/task receipt, objective τ-bench scoring, hashes,
usage/cost, and a clean credential scan, regardless of which hypothesis wins.

The completed canonical run is
`runs/exp2-4-kimi-k3-20260730-v7`: all 60 cells have real Kimi K3 action/user
receipts and no transport or task errors. Its observed pass counts were
baseline 7/10, Trump 6/10, casual 9/10, randomized organization 8/10,
no-description 9/10, and all ablations 8/10. These results complete the
preregistered experiment but do **not** reproduce the manuscript's historical
“over 30%” and “45%” point estimates; `comparison.json` records that
qualification instead of retrofitting a favorable claim.

Example **real smoke** table (`--model gpt-4o --env airline --end-index 4`, only 4 tasks/group—illustrates table shape, not stable science):

```
Experiment                        Success Rate      Tasks        Relative
----------------------------------------------------------------------
wiki_random                      50.0%         2/  4      200.0%
baseline                         25.0%         1/  4      100.0%  ⭐
tone_trump                       25.0%         1/  4      100.0%
tone_casual                      25.0%         1/  4      100.0%
no_tool_desc                      0.0%         0/  4        0.0%
all_ablations                     0.0%         0/  4        0.0%
```

> ⚠️ n=4 per arm is very noisy—e.g. `wiki_random` above baseline is chance, not a real finding. Directional signals (no tool desc → 0%, full stack → 0%, tone little effect on success) match 实验 2-4; for stable numbers use `--end-index` ≥ 10 and multiple `--seed`. Use **your** full runs, not these smoke digits.

#### Baseline (single config)

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --task-split test \
    --start-index 0 \
    --end-index 10
# bare ids → OpenAI direct; ids with '/' → openrouter
```

#### Tone ablations

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style trump \
    --ablation-name trump_tone

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --ablation-name casual_tone
```

#### Wiki randomization

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --randomize-wiki \
    --ablation-name wiki_random
```

#### Remove tool descriptions

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --remove-tool-descriptions \
    --ablation-name no_tool_desc
```

#### Combined ablations

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --randomize-wiki \
    --remove-tool-descriptions \
    --ablation-name full_ablation
```

### Experiment scripts

Two equivalent ways to run the full suite:

1. **Python one-shot (recommended):** `python run_ablation.py --env airline --end-index 10 --all`  
2. **Bash orchestration:** `run_full_ablation.sh` calls `run_ablation.py` then `analyze_results.py`:

```bash
./run_full_ablation.sh --model gpt-5.6-luna --env airline --num-tasks 10
./run_full_ablation.sh --quick   # 3 tasks/arm smoke
```

### Result analysis

Raw trajectories land in `results_ablation/` with:

- **task_id**, **reward** (0/1), **info**, **traj**, **ablation_config**

```bash
python analyze_results.py
python analyze_results.py --results-dir results_ablation --output summary.json
```

> `--all` already prints the comparison table; `analyze_results.py` is for re-aggregating historical/manual runs. Bundled `results_ablation/*.json` are small debug samples (1–6 tasks)—**not** enough for statistical claims; use full runs (`--end-index` ≥ 10).

### Expected ranking

1. **Baseline** — best  
2. **Tone variants** — usually little success-rate impact  
3. **Wiki randomization** — hurts instruction following  
4. **No tool descriptions** — many bad tool args / wrong ops  
5. **Combined** — worst  

### Key insights

Treat the Agent as a smart new employee:

1. **Clear instructions matter** — structure, task description, tool how-to  
2. **Context organization matters** — logical order, group related rules, explicit priority  
3. **Tool docs are required** — purpose, parameters, examples  

### Parameters

| Parameter | Description | Options |
|------|------|------|
| `--tone-style` | Axis 1: tone on system prompt | default, trump, casual |
| `--randomize-wiki` | Axis 2: scramble wiki structure | flag |
| `--remove-tool-descriptions` | Axis 3: strip tool docs | flag |
| `--all` | Full ablation suite + comparison table | flag |
| `--output` | Summary JSON path (`--all` only) | string |
| `--ablation-name` | Run label | string |
| `--env` | Environment | airline, retail |
| `--model` | Model id | e.g. gpt-4o-mini, gpt-4o |
| `--model-provider` | Provider (optional) | auto: bare → openai, `/` → openrouter |
| `--task-split` | Split | train, test, dev |
| `--start-index` / `--end-index` | Task range | integers |
| `--log-dir` | Results directory | string |

### Troubleshooting

1. **ImportError** — correct cwd + install deps  
2. **API errors** — keys and quota  
3. **Memory** — lower `--max-concurrency`  

Debug:

```bash
export LITELLM_LOG=DEBUG
python run_ablation.py ...
```

### Summary

Ablations quantify prompt quality: poor structure/docs can cost **30–80%** performance. Structure and clarity dominate; professionalism and consistency support effective Agents. Good prompt engineering ≈ good employee training.

### Upstream τ-bench (bundled)

This tree vendors τ-bench (tool-agent-user interaction benchmark). Upstream news: [τ²-bench](https://github.com/sierra-research/tau2-bench) adds fixes + a `telecom` domain.

**Papers:** [τ-bench](https://arxiv.org/abs/2406.12045), [τ²-Bench](https://arxiv.org/abs/2506.07982)

**Vanilla τ-bench run** (non-ablation path):

```bash
python run.py --agent-strategy tool-calling --env retail --model gpt-4o \
  --model-provider openai --user-model gpt-4o --user-model-provider openai \
  --user-strategy llm --max-concurrency 10
# optional: --task-ids 2 4 6
```

User strategies include `llm`, `react`, `verify`, `reflection`. See original τ-bench docs for leaderboards, auto error identification, and historical trajectories. License: `./LICENSE`.

---

## 中文

### 概述

扩展 [τ-bench](https://arxiv.org/abs/2406.12045) 框架，增加三个关键消融维度，演示**提示工程：把 Agent 看成聪明的新员工**的重要性，并量化语气、指令组织、工具描述对任务成功率的影响。

### 消融研究选项

#### 1. 语气风格

- **default**：标准专业语气（基线）  
- **trump**：夸张、重复强调、自信表述  
- **casual**：表情符号、俚语、轻松口吻  

**原理：** 语气影响专业性与任务质量；过于随意或夸张可能降低信任、增加误解、损害执行准确度。

#### 2. Wiki 规则随机化

使用预生成的极度混乱版 wiki：

- 移除章节标题与结构  
- 每条规则加操作上下文前缀（如 “When booking flights”）  
- 打乱成平面列表  
- 破坏规则间逻辑关系  

**原理：** 组织良好的指令像培训手册；极度随机化破坏层级、混淆规则边界、抬高误用与遗漏风险。

#### 3. 工具描述移除

- 工具与参数描述置空  
- 检验「写清楚怎么用」的重要性  

**原理：** 清晰工具说明像操作手册；去掉后误用上升、完成率下降。

### 安装

```bash
# 在仓库根目录使用统一的第 2 章环境
uv sync --locked --python 3.12 --extra ch2

# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat

# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch2]"

cd chapter2/prompt-engineering

# 迁移期间仍支持单项目兼容路径：
# python -m pip install -r requirements.txt
```

（旧文档可能写 `projects/week2/prompt-engineering`；请使用本仓库路径。）

### 使用方法

入口脚本均提供中文 `--help`：`python run_ablation.py --help`、`python analyze_results.py --help`。

#### 一键完整消融并输出对比表（推荐）

`--all` 在同一进程内依次跑基线 + 三个维度单独消融 + 全部叠加，打印成功率对比表，汇总写入 `--output`（默认 `log-dir/ablation_summary_<时间戳>.json`）。复现书中实验 2-4 最直接：

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --end-index 10 \
    --all
# 默认 OpenAI 直连（provider=openai），需 OPENAI_API_KEY。
# 走 OpenRouter：模型写成带斜杠 id（如 openai/gpt-5），需 OPENROUTER_API_KEY。
# 通用回退：裸 id（如 gpt-4o-mini）且未设 OPENAI_API_KEY、已设 OPENROUTER_API_KEY 时，
#           自动前缀为 openai/gpt-4o-mini 并切到 openrouter。
```

**真实冒烟**表示例（`--model gpt-4o --env airline --end-index 4`，每组仅 4 任务，只用于展示表格形态）：

```
Experiment                        Success Rate      Tasks        Relative
----------------------------------------------------------------------
wiki_random                      50.0%         2/  4      200.0%
baseline                         25.0%         1/  4      100.0%  ⭐
tone_trump                       25.0%         1/  4      100.0%
tone_casual                      25.0%         1/  4      100.0%
no_tool_desc                      0.0%         0/  4        0.0%
all_ablations                     0.0%         0/  4        0.0%
```

> ⚠️ 每组 4 任务噪声极大——例如 `wiki_random` 偶然高于 baseline 不是真实结论。方向性信号（去掉工具描述 → 0%、全部叠加 → 0%、语气对成功率影响小）与实验 2-4 一致；要稳定量化请把 `--end-index` 提到 10 以上并多跑 `--seed`。以你自己的完整运行为准。

#### 基线（单配置）

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --task-split test \
    --start-index 0 \
    --end-index 10
# 裸 id → OpenAI 直连；带 / 的 id → openrouter
```

#### 语气消融

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style trump \
    --ablation-name trump_tone

python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --ablation-name casual_tone
```

#### Wiki 随机化

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --randomize-wiki \
    --ablation-name wiki_random
```

#### 移除工具描述

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --remove-tool-descriptions \
    --ablation-name no_tool_desc
```

#### 组合消融

```bash
python run_ablation.py \
    --model gpt-5.6-luna \
    --env airline \
    --tone-style casual \
    --randomize-wiki \
    --remove-tool-descriptions \
    --ablation-name full_ablation
```

### 实验脚本

完整套消融有两种等价方式：

1. **Python 一键（推荐）：** `python run_ablation.py --env airline --end-index 10 --all`  
2. **Bash 编排：** `run_full_ablation.sh` 逐个调用 `run_ablation.py` 再 `analyze_results.py`：

```bash
./run_full_ablation.sh --model gpt-5.6-luna --env airline --num-tasks 10
./run_full_ablation.sh --quick   # 每组 3 任务冒烟
```

### 结果分析

原始轨迹在 `results_ablation/`，含 **task_id**、**reward**（0/1）、**info**、**traj**、**ablation_config**。

```bash
python analyze_results.py
python analyze_results.py --results-dir results_ablation --output summary.json
```

> `--all` 结束时已打印对比表；`analyze_results.py` 用于事后重汇总。仓库内 `results_ablation/*.json` 为少量调试样本（1–6 任务），**不足以做统计结论**；请用完整运行（`--end-index` ≥ 10）。

### 预期排序

1. **Baseline** — 最佳  
2. **语气变化** — 通常对成功率影响不大  
3. **Wiki 随机化** — 严重损害指令遵循  
4. **无工具描述** — 大量参数错误 / 错误操作  
5. **组合消融** — 最差  

### 关键洞察

把 Agent 看成聪明的新员工：

1. **清晰指令至关重要** — 结构化信息、任务描述、工具用法  
2. **上下文组织影响理解** — 逻辑排序、相关规则归并、优先级明确  
3. **工具文档不可或缺** — 用途、参数、示例  

### 参数说明

| 参数 | 说明 | 选项 |
|------|------|------|
| `--tone-style` | 维度一·语气风格 | default, trump, casual |
| `--randomize-wiki` | 维度二·随机化 wiki 结构 | flag |
| `--remove-tool-descriptions` | 维度三·移除工具描述 | flag |
| `--all` | 一键完整消融并打印对比表 | flag |
| `--output` | （仅 --all）汇总 JSON 路径 | string |
| `--ablation-name` | 实验名称标识 | string |
| `--env` | 环境 | airline, retail |
| `--model` | 模型 | 如 gpt-4o-mini, gpt-4o |
| `--model-provider` | 提供商（可选） | 自动：裸 id → openai，带 / → openrouter |
| `--task-split` | 任务集 | train, test, dev |
| `--start-index` / `--end-index` | 任务区间 | 整数 |
| `--log-dir` | 结果目录 | string |

### 故障排除

1. **ImportError**：确认目录与依赖  
2. **API 错误**：密钥与配额  
3. **内存**：降低 `--max-concurrency`  

```bash
export LITELLM_LOG=DEBUG
python run_ablation.py ...
```

### 总结

消融框架量化展示：提示工程不当时可出现 **30–80%** 的性能下滑；**结构与清晰度**最关键；专业性与一致性支撑有效 Agent 系统。记住：优秀的提示工程就是优秀的员工培训。

### 上游 τ-bench（内嵌）

本目录内嵌 τ-bench（工具-Agent-用户交互基准）。上游进展：[τ²-bench](https://github.com/sierra-research/tau2-bench) 含修复与 `telecom` 域。

**论文：** [τ-bench](https://arxiv.org/abs/2406.12045)、[τ²-Bench](https://arxiv.org/abs/2506.07982)

**原版（非消融）运行：**

```bash
python run.py --agent-strategy tool-calling --env retail --model gpt-4o \
  --model-provider openai --user-model gpt-4o --user-model-provider openai \
  --user-strategy llm --max-concurrency 10
# 可选：--task-ids 2 4 6
```

用户模拟策略含 `llm`、`react`、`verify`、`reflection`。排行榜、自动错误识别、历史轨迹等见原版 τ-bench 文档。许可：`./LICENSE`。

---

## Notes / 说明

- Book experiment path is `run_ablation.py`; vanilla `run.py` is the upstream τ-bench entry.  
- 书中实验主路径是 `run_ablation.py`；`run.py` 为上游 τ-bench 原版入口。  
- Smoke tables in this README are not publishable success rates.  
- 文中冒烟表不可当作可发表的成功率数字。