ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter7/model-action-threshold/README.md

@@ -0,0 +1,79 @@
+# Experiment 7-8: Model action thresholds in a fixed coding harness
+
+This experiment tests whether an explore-first or implement-first tendency
+follows the **model** when the coding harness is held fixed. Both model
+families receive the same system prompt, user task, repository, tool names,
+JSON schemas, tool results, turn limit, and independent test command. By
+default both are also routed through the same OpenRouter OpenAI-compatible
+endpoint, reducing provider-adapter differences.
+
+The neutral prompt does not require the model to read any number of files,
+produce a plan, edit early, or run tests. The experiment records what the
+model chooses to do.
+
+## Tasks and metrics
+
+Three miniature repositories cover a localized bug, a cross-cutting identity
+change, and a public-contract-sensitive cache fix. Every fixture starts with
+failing tests. Each run is performed in a fresh temporary copy and is
+independently tested at the end.
+
+Primary process metrics:
+
+- tool calls and elapsed time before the first edit;
+- read/search calls and unique files read before the first edit;
+- whether the first model-triggered test run passes;
+- edits after the first test, total edits, and files changed;
+- final test success, latency, and token usage.
+
+Time to first edit is not a quality score. Interpret it together with
+first-patch acceptance, rework, final success, and total cost.
+
+## Install and run
+
+From the repository root:
+
+```bash
+uv sync --locked --extra ch6
+export OPENROUTER_API_KEY=...
+uv run python chapter7/model-action-threshold/experiment.py \
+  --models openai/gpt-5.6-sol anthropic/claude-sonnet-5 \
+  --trials 3 \
+  --policy neutral \
+  --output chapter7/model-action-threshold/results/my-run
+```
+
+The runner alternates model order between trials and checkpoints the campaign
+after every cell. Re-running the same command and output directory resumes
+only the missing model × task × trial cells. `config.json` hashes the system prompt and tool schema;
+`observations.jsonl` retains every trajectory; `summary.json` aggregates the
+metrics; and `manifest.json` hashes those three artifacts.
+
+Run the optional harness ablation separately:
+
+```bash
+uv run python chapter7/model-action-threshold/experiment.py \
+  --models openai/gpt-5.6-sol anthropic/claude-sonnet-5 \
+  --trials 3 --policy explore-first \
+  --output chapter7/model-action-threshold/results/explore-first
+```
+
+Do not merge neutral and explore-first observations into one model comparison.
+The first run estimates the model effect under a neutral harness; comparing
+the two campaigns estimates how much an explicit harness instruction modifies
+that behavior.
+
+## Validate the implementation
+
+The offline tests verify path confinement, event-boundary accounting, rework
+measurement, aggregation, and that every fixture starts in the intended
+failing state:
+
+```bash
+python -m unittest discover -s chapter7/model-action-threshold/tests -v
+```
+
+The saved validation campaign in `results/` is considered complete only when
+its manifest contains every requested model × task × trial observation and no
+API errors. Model task failures remain valid experimental outcomes and are not
+silently discarded.

chapter7/model-action-threshold/experiment.py

@@ -0,0 +1,718 @@
+#!/usr/bin/env python3
+"""Controlled model action-threshold experiment for Chapter 6.
+
+The experiment holds the coding harness, prompt, tools, task repositories, and
+sampling order fixed while swapping only the model identifier.  It measures
+how much evidence a model gathers before its first edit and whether acting
+earlier or later leads to a correct, low-rework solution.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import shutil
+import statistics
+import subprocess
+import tempfile
+import time
+from dataclasses import asdict, dataclass, field
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+try:
+    from dotenv import load_dotenv
+
+    load_dotenv()
+except ImportError:
+    pass
+
+
+ROOT = Path(__file__).resolve().parent
+TASKS_ROOT = ROOT / "tasks"
+DEFAULT_MODELS = ["openai/gpt-5.6-sol", "anthropic/claude-sonnet-5"]
+BASE_URL = "https://openrouter.ai/api/v1"
+EDIT_TOOLS = {"replace_text", "write_file"}
+IGNORED_PARTS = {".git", "__pycache__", ".pytest_cache"}
+
+NEUTRAL_SYSTEM_PROMPT = """You are a coding agent working in a small repository.
+Complete the user's task using the provided repository tools. Keep the change
+scoped, preserve public behavior not mentioned in the task, and use tests when
+they help you validate the result. When the task is complete, respond with a
+concise summary. The harness will independently run the test command. Do not
+ask the user to perform any steps."""
+
+EXPLORE_SYSTEM_SUFFIX = """
+
+Before editing, inspect the repository structure, relevant implementation,
+callers, and tests so that you understand the full impact of the change."""
+
+
+TOOLS: list[dict[str, Any]] = [
+    {
+        "type": "function",
+        "function": {
+            "name": "list_files",
+            "description": "List repository files under a directory.",
+            "parameters": {
+                "type": "object",
+                "properties": {"path": {"type": "string", "default": "."}},
+                "additionalProperties": False,
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "read_file",
+            "description": "Read a UTF-8 text file with line numbers.",
+            "parameters": {
+                "type": "object",
+                "properties": {"path": {"type": "string"}},
+                "required": ["path"],
+                "additionalProperties": False,
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "search",
+            "description": "Search for a literal string in repository text files.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "query": {"type": "string"},
+                    "path": {"type": "string", "default": "."},
+                },
+                "required": ["query"],
+                "additionalProperties": False,
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "replace_text",
+            "description": "Replace one exact text block in an existing UTF-8 file.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "path": {"type": "string"},
+                    "old_text": {"type": "string"},
+                    "new_text": {"type": "string"},
+                },
+                "required": ["path", "old_text", "new_text"],
+                "additionalProperties": False,
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "write_file",
+            "description": "Create a new UTF-8 text file. Refuses to overwrite an existing file.",
+            "parameters": {
+                "type": "object",
+                "properties": {
+                    "path": {"type": "string"},
+                    "content": {"type": "string"},
+                },
+                "required": ["path", "content"],
+                "additionalProperties": False,
+            },
+        },
+    },
+    {
+        "type": "function",
+        "function": {
+            "name": "run_tests",
+            "description": "Run the task's fixed test command and return its output.",
+            "parameters": {"type": "object", "properties": {}, "additionalProperties": False},
+        },
+    },
+]
+
+
+def utc_now() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for block in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(block)
+    return digest.hexdigest()
+
+
+def load_task(task_id: str) -> dict[str, Any]:
+    task_dir = TASKS_ROOT / task_id
+    metadata = json.loads((task_dir / "task.json").read_text(encoding="utf-8"))
+    metadata["task_id"] = task_id
+    metadata["source_repo"] = task_dir / "repo"
+    return metadata
+
+
+def discover_tasks() -> list[str]:
+    return sorted(path.parent.name for path in TASKS_ROOT.glob("*/task.json"))
+
+
+def safe_path(repo: Path, relative: str) -> Path:
+    candidate = (repo / relative).resolve()
+    root = repo.resolve()
+    if candidate != root and root not in candidate.parents:
+        raise ValueError(f"path escapes repository: {relative}")
+    return candidate
+
+
+def visible_files(root: Path) -> list[Path]:
+    return [
+        path
+        for path in sorted(root.rglob("*"))
+        if path.is_file() and not any(part in IGNORED_PARTS for part in path.parts)
+    ]
+
+
+def snapshot(repo: Path) -> dict[str, str]:
+    return {
+        str(path.relative_to(repo)): sha256_file(path)
+        for path in visible_files(repo)
+    }
+
+
+def changed_files(before: dict[str, str], repo: Path) -> list[str]:
+    after = snapshot(repo)
+    return sorted(
+        path for path in set(before) | set(after) if before.get(path) != after.get(path)
+    )
+
+
+def run_test_command(repo: Path, command: list[str], timeout: int = 30) -> dict[str, Any]:
+    started = time.monotonic()
+    try:
+        completed = subprocess.run(
+            command,
+            cwd=repo,
+            text=True,
+            capture_output=True,
+            timeout=timeout,
+            env={**os.environ, "PYTHONDONTWRITEBYTECODE": "1"},
+            check=False,
+        )
+        output = (completed.stdout + completed.stderr)[-12000:]
+        return {
+            "passed": completed.returncode == 0,
+            "returncode": completed.returncode,
+            "duration_s": round(time.monotonic() - started, 4),
+            "output": output,
+        }
+    except subprocess.TimeoutExpired as exc:
+        return {
+            "passed": False,
+            "returncode": None,
+            "duration_s": round(time.monotonic() - started, 4),
+            "output": f"test timeout: {exc}",
+        }
+
+
+@dataclass
+class Usage:
+    input_tokens: int = 0
+    cached_input_tokens: int = 0
+    output_tokens: int = 0
+    reasoning_tokens: int = 0
+
+    def add_response(self, response: Any) -> None:
+        raw = getattr(response, "usage", None)
+        if raw is None:
+            return
+        self.input_tokens += int(getattr(raw, "prompt_tokens", 0) or 0)
+        self.output_tokens += int(getattr(raw, "completion_tokens", 0) or 0)
+        prompt_details = getattr(raw, "prompt_tokens_details", None)
+        completion_details = getattr(raw, "completion_tokens_details", None)
+        self.cached_input_tokens += int(getattr(prompt_details, "cached_tokens", 0) or 0)
+        self.reasoning_tokens += int(getattr(completion_details, "reasoning_tokens", 0) or 0)
+
+
+@dataclass
+class TraceState:
+    started: float
+    events: list[dict[str, Any]] = field(default_factory=list)
+    first_edit_sequence: int | None = None
+    first_edit_elapsed_s: float | None = None
+    first_successful_edit_sequence: int | None = None
+    first_successful_edit_elapsed_s: float | None = None
+    first_patch_test_passed: bool | None = None
+    edits_after_first_test: int = 0
+    tests_after_edit: int = 0
+    usage: Usage = field(default_factory=Usage)
+
+    def log(self, event_type: str, **payload: Any) -> dict[str, Any]:
+        event = {
+            "sequence": len(self.events) + 1,
+            "elapsed_s": round(time.monotonic() - self.started, 4),
+            "type": event_type,
+            **payload,
+        }
+        self.events.append(event)
+        return event
+
+    def record_tool(self, name: str, arguments: dict[str, Any], result: dict[str, Any]) -> None:
+        event = self.log("tool", tool=name, arguments=arguments, result=result)
+        if name in EDIT_TOOLS:
+            if self.first_edit_sequence is None:
+                self.first_edit_sequence = event["sequence"]
+                self.first_edit_elapsed_s = event["elapsed_s"]
+            if result.get("ok") and self.first_successful_edit_sequence is None:
+                self.first_successful_edit_sequence = event["sequence"]
+                self.first_successful_edit_elapsed_s = event["elapsed_s"]
+            elif result.get("ok") and self.first_patch_test_passed is not None:
+                self.edits_after_first_test += 1
+        elif name == "run_tests" and self.first_successful_edit_sequence is not None:
+            self.tests_after_edit += 1
+            if self.first_patch_test_passed is None:
+                self.first_patch_test_passed = bool(result.get("passed"))
+
+
+class RepositoryHarness:
+    def __init__(self, repo: Path, test_command: list[str], trace: TraceState):
+        self.repo = repo
+        self.test_command = test_command
+        self.trace = trace
+
+    def execute(self, name: str, arguments: dict[str, Any]) -> dict[str, Any]:
+        try:
+            if name == "list_files":
+                base = safe_path(self.repo, arguments.get("path", "."))
+                if not base.exists():
+                    result = {"error": "path does not exist"}
+                else:
+                    files = [str(path.relative_to(self.repo)) for path in visible_files(base)]
+                    result = {"files": files[:500], "count": len(files)}
+            elif name == "read_file":
+                path = safe_path(self.repo, arguments["path"])
+                text = path.read_text(encoding="utf-8")
+                numbered = "\n".join(
+                    f"{index:4d}: {line}" for index, line in enumerate(text.splitlines(), 1)
+                )
+                result = {"path": str(path.relative_to(self.repo)), "content": numbered[:30000]}
+            elif name == "search":
+                base = safe_path(self.repo, arguments.get("path", "."))
+                query = arguments["query"]
+                matches: list[str] = []
+                for path in visible_files(base):
+                    try:
+                        lines = path.read_text(encoding="utf-8").splitlines()
+                    except UnicodeDecodeError:
+                        continue
+                    for line_number, line in enumerate(lines, 1):
+                        if query in line:
+                            matches.append(
+                                f"{path.relative_to(self.repo)}:{line_number}:{line}"
+                            )
+                result = {"matches": matches[:200], "count": len(matches)}
+            elif name == "replace_text":
+                path = safe_path(self.repo, arguments["path"])
+                text = path.read_text(encoding="utf-8")
+                old = arguments["old_text"]
+                count = text.count(old)
+                if count != 1:
+                    result = {"error": f"old_text occurs {count} times; expected exactly once"}
+                else:
+                    path.write_text(text.replace(old, arguments["new_text"], 1), encoding="utf-8")
+                    result = {"ok": True, "path": str(path.relative_to(self.repo))}
+            elif name == "write_file":
+                path = safe_path(self.repo, arguments["path"])
+                if path.exists():
+                    result = {"error": "file already exists"}
+                else:
+                    path.parent.mkdir(parents=True, exist_ok=True)
+                    path.write_text(arguments["content"], encoding="utf-8")
+                    result = {"ok": True, "path": str(path.relative_to(self.repo))}
+            elif name == "run_tests":
+                result = run_test_command(self.repo, self.test_command)
+            else:
+                result = {"error": f"unknown tool: {name}"}
+        except (KeyError, OSError, UnicodeError, ValueError) as exc:
+            result = {"error": f"{type(exc).__name__}: {exc}"}
+        self.trace.record_tool(name, arguments, result)
+        return result
+
+
+def pre_edit_metrics(events: list[dict[str, Any]], first_edit_sequence: int | None) -> dict[str, Any]:
+    boundary = first_edit_sequence if first_edit_sequence is not None else float("inf")
+    tools = [event for event in events if event["type"] == "tool" and event["sequence"] < boundary]
+    reads = [event for event in tools if event["tool"] == "read_file"]
+    searches = [event for event in tools if event["tool"] == "search"]
+    files = {
+        event["result"].get("path")
+        for event in reads
+        if event["result"].get("path") is not None
+    }
+    return {
+        "tool_calls_before_first_edit": len(tools),
+        "read_calls_before_first_edit": len(reads),
+        "search_calls_before_first_edit": len(searches),
+        "unique_files_read_before_first_edit": len(files),
+        "files_read_before_first_edit": sorted(files),
+    }
+
+
+def call_model(client: Any, model: str, messages: list[dict[str, Any]]) -> Any:
+    return client.chat.completions.create(
+        model=model,
+        messages=messages,
+        tools=TOOLS,
+        tool_choice="auto",
+        max_tokens=4096,
+    )
+
+
+def run_cell(
+    client: Any,
+    model: str,
+    task: dict[str, Any],
+    trial: int,
+    policy: str,
+    max_turns: int,
+) -> dict[str, Any]:
+    started_wall = utc_now()
+    started = time.monotonic()
+    trace = TraceState(started=started)
+    with tempfile.TemporaryDirectory(prefix="action-threshold-") as temp_dir:
+        repo = Path(temp_dir) / "repo"
+        shutil.copytree(task["source_repo"], repo)
+        # macOS exposes /var through a /private/var symlink. Resolve once so
+        # path-confinement and relative-path reporting use the same root.
+        repo = repo.resolve()
+        baseline = snapshot(repo)
+        baseline_test = run_test_command(repo, task["test_command"])
+        system_prompt = NEUTRAL_SYSTEM_PROMPT
+        if policy == "explore-first":
+            system_prompt += EXPLORE_SYSTEM_SUFFIX
+        messages: list[dict[str, Any]] = [
+            {"role": "system", "content": system_prompt},
+            {"role": "user", "content": task["instruction"]},
+        ]
+        run_error: str | None = None
+        final_text = ""
+
+        for turn in range(1, max_turns + 1):
+            try:
+                response = call_model(client, model, messages)
+            except Exception as exc:  # API failures are experiment observations.
+                run_error = f"{type(exc).__name__}: {exc}"
+                trace.log("api_error", turn=turn, error=run_error)
+                break
+            trace.usage.add_response(response)
+            message = response.choices[0].message
+            final_text = message.content or ""
+            tool_calls = message.tool_calls or []
+            assistant_payload: dict[str, Any] = {
+                "role": "assistant",
+                "content": message.content or "",
+            }
+            if tool_calls:
+                assistant_payload["tool_calls"] = [
+                    {
+                        "id": call.id,
+                        "type": "function",
+                        "function": {
+                            "name": call.function.name,
+                            "arguments": call.function.arguments,
+                        },
+                    }
+                    for call in tool_calls
+                ]
+            messages.append(assistant_payload)
+            trace.log(
+                "assistant",
+                turn=turn,
+                text=message.content or "",
+                tool_names=[call.function.name for call in tool_calls],
+            )
+            if not tool_calls:
+                break
+            harness = RepositoryHarness(repo, task["test_command"], trace)
+            for tool_call in tool_calls:
+                try:
+                    arguments = json.loads(tool_call.function.arguments or "{}")
+                except json.JSONDecodeError as exc:
+                    arguments = {}
+                    result = {"error": f"invalid tool JSON: {exc}"}
+                    trace.record_tool(tool_call.function.name, arguments, result)
+                else:
+                    result = harness.execute(tool_call.function.name, arguments)
+                messages.append(
+                    {
+                        "role": "tool",
+                        "tool_call_id": tool_call.id,
+                        "content": json.dumps(result, ensure_ascii=False),
+                    }
+                )
+        else:
+            run_error = f"maximum turns reached ({max_turns})"
+
+        final_test = run_test_command(repo, task["test_command"])
+        changed = changed_files(baseline, repo)
+        edits = [
+            event for event in trace.events
+            if event["type"] == "tool" and event["tool"] in EDIT_TOOLS
+        ]
+        successful_edits = [event for event in edits if event["result"].get("ok")]
+        metrics = pre_edit_metrics(trace.events, trace.first_edit_sequence)
+        return {
+            "schema_version": 1,
+            "model": model,
+            "task_id": task["task_id"],
+            "task_category": task["category"],
+            "trial": trial,
+            "policy": policy,
+            "started_at_utc": started_wall,
+            "duration_s": round(time.monotonic() - started, 4),
+            "baseline_test_passed": baseline_test["passed"],
+            "baseline_test_returncode": baseline_test["returncode"],
+            "run_error": run_error,
+            "first_edit_sequence": trace.first_edit_sequence,
+            "seconds_to_first_edit": trace.first_edit_elapsed_s,
+            "first_successful_edit_sequence": trace.first_successful_edit_sequence,
+            "seconds_to_first_successful_edit": trace.first_successful_edit_elapsed_s,
+            **metrics,
+            "first_patch_test_passed": trace.first_patch_test_passed,
+            "edit_attempts_total": len(edits),
+            "successful_edit_calls_total": len(successful_edits),
+            "edits_after_first_test": trace.edits_after_first_test,
+            "tests_after_edit": trace.tests_after_edit,
+            "changed_files": changed,
+            "changed_file_count": len(changed),
+            "final_test_passed": final_test["passed"],
+            "final_test_returncode": final_test["returncode"],
+            "final_test_output": final_test["output"],
+            "final_text": final_text,
+            "usage": asdict(trace.usage),
+            "events": trace.events,
+        }
+
+
+def mean_or_none(values: list[float | int | None]) -> float | None:
+    cleaned = [float(value) for value in values if value is not None]
+    return round(statistics.mean(cleaned), 4) if cleaned else None
+
+
+def summarize(observations: list[dict[str, Any]]) -> dict[str, Any]:
+    models = sorted({row["model"] for row in observations})
+    by_model: list[dict[str, Any]] = []
+    for model in models:
+        rows = [row for row in observations if row["model"] == model]
+        tested_first_patches = [
+            row for row in rows if row["first_patch_test_passed"] is not None
+        ]
+        by_model.append(
+            {
+                "model": model,
+                "runs": len(rows),
+                "completed_without_api_error": sum(row["run_error"] is None for row in rows),
+                "final_pass_rate": round(sum(row["final_test_passed"] for row in rows) / len(rows), 4),
+                "first_patch_pass_rate": (
+                    round(
+                        sum(row["first_patch_test_passed"] for row in tested_first_patches)
+                        / len(tested_first_patches),
+                        4,
+                    )
+                    if tested_first_patches else None
+                ),
+                "mean_tool_calls_before_first_edit": mean_or_none(
+                    [row["tool_calls_before_first_edit"] for row in rows]
+                ),
+                "mean_unique_files_read_before_first_edit": mean_or_none(
+                    [row["unique_files_read_before_first_edit"] for row in rows]
+                ),
+                "mean_seconds_to_first_edit": mean_or_none(
+                    [row["seconds_to_first_edit"] for row in rows]
+                ),
+                "mean_edit_attempts": mean_or_none([row["edit_attempts_total"] for row in rows]),
+                "mean_successful_edit_calls": mean_or_none(
+                    [row["successful_edit_calls_total"] for row in rows]
+                ),
+                "mean_edits_after_first_test": mean_or_none(
+                    [row["edits_after_first_test"] for row in rows]
+                ),
+                "mean_changed_files": mean_or_none([row["changed_file_count"] for row in rows]),
+                "total_input_tokens": sum(row["usage"]["input_tokens"] for row in rows),
+                "total_output_tokens": sum(row["usage"]["output_tokens"] for row in rows),
+            }
+        )
+    return {
+        "schema_version": 1,
+        "generated_at_utc": utc_now(),
+        "observation_count": len(observations),
+        "by_model": by_model,
+        "by_task": [
+            {
+                "task_id": task_id,
+                "models": [
+                    {
+                        "model": model,
+                        "runs": len(rows := [
+                            row for row in observations
+                            if row["task_id"] == task_id and row["model"] == model
+                        ]),
+                        "final_pass_rate": (
+                            round(sum(row["final_test_passed"] for row in rows) / len(rows), 4)
+                            if rows else None
+                        ),
+                        "mean_files_read_before_edit": mean_or_none(
+                            [row["unique_files_read_before_first_edit"] for row in rows]
+                        ),
+                    }
+                    for model in models
+                ],
+            }
+            for task_id in sorted({row["task_id"] for row in observations})
+        ],
+    }
+
+
+def write_json(path: Path, value: Any) -> None:
+    path.write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
+
+
+def write_campaign(
+    output_dir: Path,
+    config: dict[str, Any],
+    observations: list[dict[str, Any]],
+) -> None:
+    output_dir.mkdir(parents=True, exist_ok=True)
+    config_path = output_dir / "config.json"
+    observations_path = output_dir / "observations.jsonl"
+    summary_path = output_dir / "summary.json"
+    write_json(config_path, config)
+    observations_path.write_text(
+        "".join(json.dumps(row, ensure_ascii=False) + "\n" for row in observations),
+        encoding="utf-8",
+    )
+    write_json(summary_path, summarize(observations))
+    artifacts = [config_path, observations_path, summary_path]
+    expected = len(config["models"]) * len(config["tasks"]) * config["trials"]
+    api_errors = sum(row["run_error"] is not None for row in observations)
+    manifest = {
+        "schema_version": 1,
+        "created_at_utc": utc_now(),
+        "status": (
+            "complete" if len(observations) == expected and api_errors == 0 else "incomplete"
+        ),
+        "expected_observations": expected,
+        "actual_observations": len(observations),
+        "api_error_count": api_errors,
+        "artifacts": {
+            path.name: {"sha256": sha256_file(path), "bytes": path.stat().st_size}
+            for path in artifacts
+        },
+    }
+    write_json(output_dir / "manifest.json", manifest)
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--models", nargs="+", default=DEFAULT_MODELS)
+    parser.add_argument("--tasks", nargs="+", default=discover_tasks())
+    parser.add_argument("--trials", type=int, default=3)
+    parser.add_argument("--policy", choices=["neutral", "explore-first"], default="neutral")
+    parser.add_argument("--max-turns", type=int, default=20)
+    parser.add_argument("--base-url", default=BASE_URL)
+    parser.add_argument("--api-key-env", default="OPENROUTER_API_KEY")
+    parser.add_argument(
+        "--output",
+        type=Path,
+        default=ROOT / "results" / f"run-{datetime.now().strftime('%Y%m%d-%H%M%S')}",
+    )
+    parser.add_argument("--list-tasks", action="store_true")
+    return parser.parse_args()
+
+
+def main() -> int:
+    args = parse_args()
+    if args.list_tasks:
+        for task_id in discover_tasks():
+            task = load_task(task_id)
+            print(f"{task_id}\t{task['category']}\t{task['instruction']}")
+        return 0
+    if args.trials < 1:
+        raise SystemExit("--trials must be at least 1")
+    unknown = sorted(set(args.tasks) - set(discover_tasks()))
+    if unknown:
+        raise SystemExit(f"unknown tasks: {', '.join(unknown)}")
+    api_key = os.getenv(args.api_key_env)
+    if not api_key:
+        raise SystemExit(f"{args.api_key_env} is not set")
+    from openai import OpenAI
+
+    client = OpenAI(api_key=api_key, base_url=args.base_url, timeout=180.0)
+    config = {
+        "schema_version": 1,
+        "created_at_utc": utc_now(),
+        "models": args.models,
+        "tasks": args.tasks,
+        "trials": args.trials,
+        "policy": args.policy,
+        "max_turns": args.max_turns,
+        "base_url": args.base_url,
+        "system_prompt_sha256": hashlib.sha256(
+            (NEUTRAL_SYSTEM_PROMPT + (EXPLORE_SYSTEM_SUFFIX if args.policy == "explore-first" else "")).encode()
+        ).hexdigest(),
+        "tool_schema_sha256": hashlib.sha256(
+            json.dumps(TOOLS, sort_keys=True).encode()
+        ).hexdigest(),
+    }
+    observations_path = args.output / "observations.jsonl"
+    observations: list[dict[str, Any]] = []
+    if observations_path.exists():
+        saved_config_path = args.output / "config.json"
+        if not saved_config_path.exists():
+            raise SystemExit("cannot resume: observations.jsonl exists without config.json")
+        saved_config = json.loads(saved_config_path.read_text(encoding="utf-8"))
+        execution_keys = {
+            "models", "tasks", "trials", "policy", "max_turns", "base_url",
+            "system_prompt_sha256", "tool_schema_sha256",
+        }
+        if any(saved_config.get(key) != config.get(key) for key in execution_keys):
+            raise SystemExit("cannot resume: saved campaign configuration does not match")
+        observations = [
+            json.loads(line) for line in observations_path.read_text(encoding="utf-8").splitlines()
+            if line.strip()
+        ]
+        config = saved_config
+        print(f"Resuming {len(observations)} saved observations from {args.output}", flush=True)
+    completed = {
+        (row["model"], row["task_id"], row["trial"], row["policy"])
+        for row in observations
+    }
+    for trial in range(1, args.trials + 1):
+        model_order = args.models if trial % 2 else list(reversed(args.models))
+        for task_id in args.tasks:
+            task = load_task(task_id)
+            for model in model_order:
+                cell = (model, task_id, trial, args.policy)
+                if cell in completed:
+                    continue
+                print(f"[{len(observations) + 1}] model={model} task={task_id} trial={trial}", flush=True)
+                row = run_cell(client, model, task, trial, args.policy, args.max_turns)
+                observations.append(row)
+                completed.add(cell)
+                print(
+                    f"    pre-edit files={row['unique_files_read_before_first_edit']} "
+                    f"tools={row['tool_calls_before_first_edit']} "
+                    f"final_pass={row['final_test_passed']} error={row['run_error']}",
+                    flush=True,
+                )
+                write_campaign(args.output, config, observations)
+    print(f"Results: {args.output}")
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter7/model-action-threshold/tasks/cache_contract/repo/cachelib/__init__.py

@@ -0,0 +1,3 @@
+from cachelib.cache import Cache
+
+__all__ = ["Cache"]

chapter7/model-action-threshold/tasks/cache_contract/repo/cachelib/cache.py

@@ -0,0 +1,17 @@
+class Cache:
+    def __init__(self):
+        self._values = {}
+
+    def get(self, key, default=None):
+        return self._values.get(key, default)
+
+    def put(self, key, value):
+        self._values[key] = value
+
+    def get_or_load(self, key, loader):
+        cached = self.get(key)
+        if cached is not None:
+            return cached
+        value = loader(key)
+        self.put(key, value)
+        return value

chapter7/model-action-threshold/tasks/cache_contract/repo/cachelib/catalog.py

@@ -0,0 +1,7 @@
+class Catalog:
+    def __init__(self, cache, fetch_product):
+        self.cache = cache
+        self.fetch_product = fetch_product
+
+    def product(self, product_id):
+        return self.cache.get_or_load(product_id, self.fetch_product)

chapter7/model-action-threshold/tasks/cache_contract/repo/tests/test_cache.py

@@ -0,0 +1,40 @@
+import unittest
+
+from cachelib import Cache
+
+
+class CacheContractTests(unittest.TestCase):
+    def test_negative_lookup_is_loaded_once(self):
+        cache = Cache()
+        calls = []
+
+        def loader(key):
+            calls.append(key)
+            return None
+
+        self.assertIsNone(cache.get_or_load("missing", loader))
+        self.assertIsNone(cache.get_or_load("missing", loader))
+        self.assertEqual(calls, ["missing"])
+
+    def test_custom_default_still_distinguishes_missing_from_none(self):
+        cache = Cache()
+        marker = object()
+        self.assertIs(cache.get("unknown", marker), marker)
+        cache.put("known-none", None)
+        self.assertIsNone(cache.get("known-none", marker))
+
+    def test_falsey_values_are_cached(self):
+        cache = Cache()
+        calls = []
+
+        def loader(key):
+            calls.append(key)
+            return 0
+
+        self.assertEqual(cache.get_or_load("zero", loader), 0)
+        self.assertEqual(cache.get_or_load("zero", loader), 0)
+        self.assertEqual(calls, ["zero"])
+
+
+if __name__ == "__main__":
+    unittest.main()

chapter7/model-action-threshold/tasks/cache_contract/task.json

@@ -0,0 +1,5 @@
+{
+  "category": "contract-sensitive",
+  "instruction": "Fix the cache so that negative lookups (a loader returning None) are cached instead of reloaded on every call. Preserve the public get(key, default=None), put, and get_or_load APIs, including the ability to distinguish a missing key when a caller supplies a custom default.",
+  "test_command": ["python", "-m", "unittest", "discover", "-s", "tests", "-v"]
+}

chapter7/model-action-threshold/tasks/crosscutting_identity/repo/accounts/__init__.py

@@ -0,0 +1 @@
+"""Small account service used by the action-threshold experiment."""

chapter7/model-action-threshold/tasks/crosscutting_identity/repo/accounts/authentication.py

@@ -0,0 +1,5 @@
+def authenticate(store, username, password):
+    profile = store.find(username.strip())
+    if profile is None or profile.password != password:
+        return None
+    return profile

chapter7/model-action-threshold/tasks/crosscutting_identity/repo/accounts/directory.py

@@ -0,0 +1,2 @@
+def lookup_profile(store, username):
+    return store.find(username)

chapter7/model-action-threshold/tasks/crosscutting_identity/repo/accounts/models.py

@@ -0,0 +1,8 @@
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class Profile:
+    username: str
+    email: str
+    password: str

chapter7/model-action-threshold/tasks/crosscutting_identity/repo/accounts/registration.py

@@ -0,0 +1,9 @@
+from accounts.models import Profile
+
+
+def register(store, username, email, password):
+    display_name = username.strip()
+    key = display_name.lower()
+    profile = Profile(display_name, email, password)
+    store.save(key, profile)
+    return profile

chapter7/model-action-threshold/tasks/crosscutting_identity/repo/accounts/store.py

@@ -0,0 +1,11 @@
+class AccountStore:
+    def __init__(self):
+        self._profiles = {}
+
+    def save(self, key, profile):
+        if key in self._profiles:
+            raise ValueError("username already exists")
+        self._profiles[key] = profile
+
+    def find(self, key):
+        return self._profiles.get(key)

chapter7/model-action-threshold/tasks/crosscutting_identity/repo/tests/test_accounts.py

@@ -0,0 +1,35 @@
+import unittest
+
+from accounts.authentication import authenticate
+from accounts.directory import lookup_profile
+from accounts.registration import register
+from accounts.store import AccountStore
+
+
+class AccountIdentityTests(unittest.TestCase):
+    def setUp(self):
+        self.store = AccountStore()
+
+    def test_registration_preserves_display_name(self):
+        profile = register(self.store, "  Alice.Dev  ", "a@example.com", "secret")
+        self.assertEqual(profile.username, "Alice.Dev")
+
+    def test_login_ignores_case_and_whitespace(self):
+        register(self.store, "Alice.Dev", "a@example.com", "secret")
+        profile = authenticate(self.store, "  ALICE.dev ", "secret")
+        self.assertIsNotNone(profile)
+        self.assertEqual(profile.username, "Alice.Dev")
+
+    def test_directory_lookup_uses_same_identity(self):
+        register(self.store, "Alice.Dev", "a@example.com", "secret")
+        self.assertIsNotNone(lookup_profile(self.store, " alice.DEV "))
+
+    def test_unicode_casefold_and_duplicate_detection(self):
+        register(self.store, "Straße", "one@example.com", "secret")
+        self.assertIsNotNone(authenticate(self.store, "STRASSE", "secret"))
+        with self.assertRaises(ValueError):
+            register(self.store, " strasse ", "two@example.com", "secret")
+
+
+if __name__ == "__main__":
+    unittest.main()

chapter7/model-action-threshold/tasks/crosscutting_identity/task.json

@@ -0,0 +1,5 @@
+{
+  "category": "cross-cutting",
+  "instruction": "Make usernames case-insensitive and insensitive to surrounding whitespace throughout registration, authentication, and directory lookup. Preserve the user's trimmed display casing in returned profiles, and handle Unicode usernames correctly. Keep the existing public APIs.",
+  "test_command": ["python", "-m", "unittest", "discover", "-s", "tests", "-v"]
+}

chapter7/model-action-threshold/tasks/localized_timeout/repo/app/__init__.py

@@ -0,0 +1 @@
+"""Example application package."""

chapter7/model-action-threshold/tasks/localized_timeout/repo/app/config.py

@@ -0,0 +1,14 @@
+"""Runtime timeout configuration."""
+
+DEFAULT_TIMEOUT = 30
+
+
+def resolve_timeout(explicit=None, env=None):
+    """Return a positive timeout using explicit > environment > default."""
+    env = env or {}
+    raw = env.get("AGENT_TIMEOUT", explicit)
+    try:
+        value = int(raw)
+    except (TypeError, ValueError):
+        return DEFAULT_TIMEOUT
+    return value if value > 0 else DEFAULT_TIMEOUT

chapter7/model-action-threshold/tasks/localized_timeout/repo/app/worker.py

@@ -0,0 +1,7 @@
+"""Worker construction kept separate from configuration parsing."""
+
+from app.config import resolve_timeout
+
+
+def worker_options(timeout=None, env=None):
+    return {"timeout": resolve_timeout(timeout, env), "retries": 2}

chapter7/model-action-threshold/tasks/localized_timeout/repo/tests/test_config.py

@@ -0,0 +1,20 @@
+import unittest
+
+from app.config import DEFAULT_TIMEOUT, resolve_timeout
+
+
+class ResolveTimeoutTests(unittest.TestCase):
+    def test_explicit_value_wins_over_environment(self):
+        self.assertEqual(resolve_timeout(12, {"AGENT_TIMEOUT": "45"}), 12)
+
+    def test_environment_is_used_without_explicit_value(self):
+        self.assertEqual(resolve_timeout(None, {"AGENT_TIMEOUT": "45"}), 45)
+
+    def test_invalid_values_use_default(self):
+        self.assertEqual(resolve_timeout("nope", {}), DEFAULT_TIMEOUT)
+        self.assertEqual(resolve_timeout(0, {}), DEFAULT_TIMEOUT)
+        self.assertEqual(resolve_timeout(-2, {}), DEFAULT_TIMEOUT)
+
+
+if __name__ == "__main__":
+    unittest.main()

chapter7/model-action-threshold/tasks/localized_timeout/task.json

@@ -0,0 +1,5 @@
+{
+  "category": "localized",
+  "instruction": "Fix the timeout configuration bug exposed by the tests. The public resolve_timeout API must remain unchanged. Explicit arguments should take precedence over AGENT_TIMEOUT, and invalid or non-positive values should fall back to the default.",
+  "test_command": ["python", "-m", "unittest", "discover", "-s", "tests", "-v"]
+}

chapter7/model-action-threshold/tests/test_experiment.py

@@ -0,0 +1,117 @@
+import importlib.util
+import sys
+import tempfile
+import unittest
+from pathlib import Path
+
+
+MODULE_PATH = Path(__file__).resolve().parents[1] / "experiment.py"
+SPEC = importlib.util.spec_from_file_location("action_threshold_experiment", MODULE_PATH)
+experiment = importlib.util.module_from_spec(SPEC)
+sys.modules[SPEC.name] = experiment
+SPEC.loader.exec_module(experiment)
+
+
+class ExperimentMechanicsTests(unittest.TestCase):
+    def test_safe_path_rejects_escape(self):
+        with tempfile.TemporaryDirectory() as temp_dir:
+            root = Path(temp_dir)
+            with self.assertRaises(ValueError):
+                experiment.safe_path(root, "../escape")
+
+    def test_trace_computes_pre_edit_boundary(self):
+        trace = experiment.TraceState(started=experiment.time.monotonic())
+        trace.record_tool("list_files", {"path": "."}, {"files": ["a.py"]})
+        trace.record_tool("read_file", {"path": "a.py"}, {"path": "a.py", "content": "x"})
+        trace.record_tool(
+            "replace_text",
+            {"path": "a.py", "old_text": "x", "new_text": "y"},
+            {"ok": True, "path": "a.py"},
+        )
+        trace.record_tool("read_file", {"path": "b.py"}, {"path": "b.py", "content": "z"})
+        metrics = experiment.pre_edit_metrics(trace.events, trace.first_edit_sequence)
+        self.assertEqual(metrics["tool_calls_before_first_edit"], 2)
+        self.assertEqual(metrics["unique_files_read_before_first_edit"], 1)
+        self.assertEqual(metrics["files_read_before_first_edit"], ["a.py"])
+
+    def test_first_patch_and_rework_are_recorded(self):
+        trace = experiment.TraceState(started=experiment.time.monotonic())
+        trace.record_tool("replace_text", {}, {"ok": True})
+        trace.record_tool("run_tests", {}, {"passed": False})
+        trace.record_tool("replace_text", {}, {"ok": True})
+        self.assertFalse(trace.first_patch_test_passed)
+        self.assertEqual(trace.edits_after_first_test, 1)
+
+    def test_each_fixture_starts_failing_and_has_safe_test_command(self):
+        for task_id in experiment.discover_tasks():
+            task = experiment.load_task(task_id)
+            self.assertEqual(task["test_command"][:4], ["python", "-m", "unittest", "discover"])
+            with tempfile.TemporaryDirectory() as temp_dir:
+                repo = Path(temp_dir) / "repo"
+                experiment.shutil.copytree(task["source_repo"], repo)
+                result = experiment.run_test_command(repo, task["test_command"])
+                self.assertFalse(result["passed"], task_id)
+
+    def test_summary_groups_models(self):
+        base = {
+            "task_id": "t",
+            "final_test_passed": True,
+            "first_patch_test_passed": True,
+            "run_error": None,
+            "tool_calls_before_first_edit": 1,
+            "unique_files_read_before_first_edit": 1,
+            "seconds_to_first_edit": 1.0,
+            "edit_attempts_total": 1,
+            "successful_edit_calls_total": 1,
+            "edits_after_first_test": 0,
+            "changed_file_count": 1,
+            "usage": {"input_tokens": 10, "output_tokens": 5},
+        }
+        summary = experiment.summarize([
+            {**base, "model": "model-a"},
+            {**base, "model": "model-b", "unique_files_read_before_first_edit": 3},
+        ])
+        self.assertEqual(summary["observation_count"], 2)
+        self.assertEqual(len(summary["by_model"]), 2)
+
+    def test_partial_summary_tolerates_missing_model_task_cell(self):
+        rows = [
+            {
+                "model": "model-a",
+                "task_id": "task-one",
+                "final_test_passed": True,
+                "first_patch_test_passed": None,
+                "run_error": None,
+                "tool_calls_before_first_edit": 1,
+                "unique_files_read_before_first_edit": 1,
+                "seconds_to_first_edit": 1.0,
+                "edit_attempts_total": 1,
+                "successful_edit_calls_total": 1,
+                "edits_after_first_test": 0,
+                "changed_file_count": 1,
+                "usage": {"input_tokens": 10, "output_tokens": 5},
+            },
+            {
+                "model": "model-b",
+                "task_id": "task-two",
+                "final_test_passed": True,
+                "first_patch_test_passed": None,
+                "run_error": None,
+                "tool_calls_before_first_edit": 1,
+                "unique_files_read_before_first_edit": 1,
+                "seconds_to_first_edit": 1.0,
+                "edit_attempts_total": 1,
+                "successful_edit_calls_total": 1,
+                "edits_after_first_test": 0,
+                "changed_file_count": 1,
+                "usage": {"input_tokens": 10, "output_tokens": 5},
+            },
+        ]
+        summary = experiment.summarize(rows)
+        missing = summary["by_task"][0]["models"][1]
+        self.assertEqual(missing["runs"], 0)
+        self.assertIsNone(missing["final_pass_rate"])
+
+
+if __name__ == "__main__":
+    unittest.main()