ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter7/tau2-bench-eval/README.md

@@ -0,0 +1,86 @@
+# Experiment 7-1: τ²-bench telecom evaluation
+
+This directory retains the bounded τ²-bench campaign requested by the
+manuscript: five telecom tasks, one trial per task, with the same model acting
+as the customer-service Agent and user simulator.
+
+## Code map
+
+- **Run first:** follow the pinned external checkout command below and run one task with num-trials 1.
+- **Start here:** the τ²-bench CLI is the runner; this directory is the reproducibility and evidence wrapper.
+- **Core behavior:** the external telecom environment executes the Agent/user turns; this project records the resulting trajectory.
+- **State / protocol:** saved raw trajectory, task seed, model IDs and run manifest under validation/runs/.
+- **Verifier:** task reward plus the chapter acceptance checks; inspect the failed task record, not only the 4/5 aggregate.
+- **Experiment variable:** fixed task set, model pair, concurrency and seed.
+- **Skip on first pass:** upstream framework internals and cost-report formatting.
+
+## Reproduction
+
+The external checkout is deliberately not vendored. Clone and pin the
+authoritative source first:
+
+```bash
+git clone https://github.com/sierra-research/tau2-bench.git chapter7/tau2-bench
+git -C chapter7/tau2-bench checkout --detach 8d005b0e5b9e4af0bc055886fa7f95fc86d1710e
+cd chapter7/tau2-bench
+uv venv --python 3.12
+uv pip install -e .
+```
+
+With `OPENROUTER_API_KEY` configured, the saved campaign used:
+
+```bash
+.venv/bin/tau2 run \
+  --domain telecom \
+  --agent-llm openrouter/openai/gpt-4.1-mini \
+  --user-llm openrouter/openai/gpt-4.1-mini \
+  --num-trials 1 \
+  --num-tasks 5 \
+  --max-concurrency 3 \
+  --save-to exp7-1-openrouter-gpt41mini-telecom-5tasks-20260802-v1 \
+  --log-level INFO
+```
+
+Both model temperatures were `0`; τ²-bench recorded seed `300`. The retained
+raw trajectory is under
+[`validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/`](validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/).
+
+## Result
+
+The Agent passed 4/5 tasks, for average reward and Pass@1 of **0.80**. All five
+simulations ended normally with `user_stop`; there were no provider errors.
+The retained provider-reported costs total about **$0.151312**: $0.112672 for
+the Agent and $0.0386396 for the user simulator.
+
+The failed task was
+`[mobile_data_issue]data_saver_mode_on|data_usage_exceeded[PERSONA:Easy]`.
+The customer supplied phone `555-123-2002`, but the Agent selected line
+`L1001`. A later `get_details_by_id(L1001)` result explicitly associated that
+line with phone `555-123-2001`; nevertheless, the Agent continued using its
+3.2/5 GB usage reading. It correctly had the user disable Data Saver, but did
+not inspect the matching `L1002` line or perform the required 2 GB data refuel.
+It spent the remainder of a 71-message trajectory on unrelated diagnostics and
+ultimately transferred to a human. Consequently, `refuel_data` and all three
+downstream environment assertions failed. The trajectory also exposes an
+earlier policy violation where the Agent emitted two customer-lookup tool calls
+in one turn even though the telecom policy permits only one at a time.
+
+This is a useful dual-control failure: the user-side Data Saver action occurred
+and was verified in the shared environment, while the Agent-side line-selection
+mistake prevented the second state mutation and final recovery.
+
+## Verification boundary
+
+The upstream public verifier reports:
+
+- format validation: passed;
+- trial-count validation: passed;
+- task validation: failed because a public leaderboard submission must cover
+  the full telecom task set.
+
+That coverage failure is expected for the five-task command specified by this
+book experiment. This evidence therefore establishes the bounded Experiment
+7-1 campaign, not a full-domain τ²-bench leaderboard result. See
+[`evidence.json`](validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/evidence.json)
+for machine-readable outcomes and [`manifest.json`](validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/manifest.json)
+for content hashes.

chapter7/tau2-bench-eval/validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/evidence.json

@@ -0,0 +1,99 @@
+{
+  "schema_version": 1,
+  "experiment": "7-1",
+  "status": "complete_bounded_campaign",
+  "generated_at_utc": "2026-08-02T09:15:41.750622Z",
+  "source": {
+    "repository": "https://github.com/sierra-research/tau2-bench",
+    "git_commit": "8d005b0e5b9e4af0bc055886fa7f95fc86d1710e"
+  },
+  "configuration": {
+    "domain": "telecom",
+    "agent_model": "openrouter/openai/gpt-4.1-mini",
+    "user_simulator_model": "openrouter/openai/gpt-4.1-mini",
+    "agent_temperature": 0.0,
+    "user_simulator_temperature": 0.0,
+    "seed": 300,
+    "num_tasks": 5,
+    "num_trials": 1,
+    "max_concurrency": 3
+  },
+  "summary": {
+    "tasks_completed": 5,
+    "tasks_passed": 4,
+    "tasks_failed": 1,
+    "average_reward": 0.8,
+    "pass_at_1": 0.8,
+    "provider_errors": 0,
+    "termination_reasons": {
+      "user_stop": 5
+    },
+    "agent_cost_usd": 0.112672,
+    "user_simulator_cost_usd": 0.0386396,
+    "total_cost_usd": 0.1513116
+  },
+  "task_outcomes": [
+    {
+      "task_id": "[mobile_data_issue]airplane_mode_on|user_abroad_roaming_enabled_off[PERSONA:None]",
+      "reward": 1.0,
+      "duration_seconds": 45.657303331419826,
+      "termination_reason": "user_stop"
+    },
+    {
+      "task_id": "[mobile_data_issue]airplane_mode_on|data_mode_off[PERSONA:None]",
+      "reward": 1.0,
+      "duration_seconds": 55.878302780911326,
+      "termination_reason": "user_stop"
+    },
+    {
+      "task_id": "[mobile_data_issue]data_mode_off|data_usage_exceeded[PERSONA:None]",
+      "reward": 1.0,
+      "duration_seconds": 64.09661028999835,
+      "termination_reason": "user_stop"
+    },
+    {
+      "task_id": "[mobile_data_issue]data_usage_exceeded|user_abroad_roaming_enabled_off[PERSONA:Easy]",
+      "reward": 1.0,
+      "duration_seconds": 45.86838342156261,
+      "termination_reason": "user_stop"
+    },
+    {
+      "task_id": "[mobile_data_issue]data_saver_mode_on|data_usage_exceeded[PERSONA:Easy]",
+      "reward": 0.0,
+      "duration_seconds": 96.15721444413066,
+      "termination_reason": "user_stop"
+    }
+  ],
+  "failed_task_analysis": {
+    "task_id": "[mobile_data_issue]data_saver_mode_on|data_usage_exceeded[PERSONA:Easy]",
+    "messages": 71,
+    "customer_phone": "555-123-2002",
+    "incorrect_line_id": "L1001",
+    "incorrect_line_phone": "555-123-2001",
+    "required_line_id": "L1002",
+    "observed_wrong_line_usage_gb": 3.2,
+    "observed_wrong_line_limit_gb": 5.0,
+    "required_refuel_gb": 2.0,
+    "passed_action_checks": [
+      "toggle_data_saver_mode"
+    ],
+    "failed_action_checks": [
+      "refuel_data"
+    ],
+    "failed_environment_assertions": [
+      "assert_mobile_data_status",
+      "assert_internet_speed",
+      "assert_data_refueling_amount"
+    ],
+    "root_cause": "The Agent failed to map the supplied phone number to its matching line, continued after tool output exposed the mismatch, and therefore missed the required data-refuel path.",
+    "secondary_policy_defect": "The Agent emitted get_customer_by_phone and get_customer_by_name in one turn despite the one-tool-call-at-a-time telecom policy."
+  },
+  "upstream_public_verification": {
+    "command": ".venv/bin/tau2 submit verify-trajs ../tau2-bench-eval/validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/trajectories.json",
+    "format_validation": "passed",
+    "trial_count_validation": "passed",
+    "task_validation": "failed_expected_bounded_campaign",
+    "task_validation_reason": "The verifier requires full-domain leaderboard coverage; this manuscript experiment deliberately runs five telecom tasks."
+  },
+  "acceptance_boundary": "Complete for the manuscript's bounded five-task Experiment 7-1 campaign; not a full-domain tau2-bench leaderboard submission."
+}

chapter7/tau2-bench-eval/validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/manifest.json

@@ -0,0 +1,29 @@
+{
+  "schema_version": 1,
+  "experiment": "7-1",
+  "status": "complete_bounded_campaign",
+  "generated_at_utc": "2026-08-02T09:15:41.750622Z",
+  "run_directory": "chapter7/tau2-bench-eval/validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1",
+  "source_repository": "https://github.com/sierra-research/tau2-bench",
+  "source_git_commit": "8d005b0e5b9e4af0bc055886fa7f95fc86d1710e",
+  "artifacts": [
+    {
+      "path": "chapter7/tau2-bench-eval/validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/trajectories.json",
+      "bytes": 386784,
+      "sha256": "22e217ca849848e4d50c3ff257bea5ce691f226066fb0af3ca3c43bdae27631b",
+      "description": "Raw upstream tau2-bench task definitions, messages, tool calls, costs, and reward records for all five simulations."
+    },
+    {
+      "path": "chapter7/tau2-bench-eval/validation/runs/exp7-1-openrouter-gpt41mini-telecom-20260802-v1/evidence.json",
+      "bytes": 3702,
+      "sha256": "4a79db9c6f73ba8c9b7c55a8e1b43b55e7a798c6c631938af39260461250aef9",
+      "description": "Machine-readable campaign summary, task outcomes, failure analysis, and verification boundary."
+    }
+  ],
+  "credential_scan": {
+    "openrouter_api_key_matches": 0,
+    "scope": "retained run directory",
+    "note": "The configured credential value was searched without printing it."
+  },
+  "acceptance_boundary": "Complete for the manuscript's bounded five-task Experiment 7-1 campaign; not a full-domain tau2-bench leaderboard submission."
+}