liqiang
b119135836
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
80 lines
3.1 KiB
Markdown
80 lines
3.1 KiB
Markdown
# Experiment 7-8: Model action thresholds in a fixed coding harness
|
||
|
||
This experiment tests whether an explore-first or implement-first tendency
|
||
follows the **model** when the coding harness is held fixed. Both model
|
||
families receive the same system prompt, user task, repository, tool names,
|
||
JSON schemas, tool results, turn limit, and independent test command. By
|
||
default both are also routed through the same OpenRouter OpenAI-compatible
|
||
endpoint, reducing provider-adapter differences.
|
||
|
||
The neutral prompt does not require the model to read any number of files,
|
||
produce a plan, edit early, or run tests. The experiment records what the
|
||
model chooses to do.
|
||
|
||
## Tasks and metrics
|
||
|
||
Three miniature repositories cover a localized bug, a cross-cutting identity
|
||
change, and a public-contract-sensitive cache fix. Every fixture starts with
|
||
failing tests. Each run is performed in a fresh temporary copy and is
|
||
independently tested at the end.
|
||
|
||
Primary process metrics:
|
||
|
||
- tool calls and elapsed time before the first edit;
|
||
- read/search calls and unique files read before the first edit;
|
||
- whether the first model-triggered test run passes;
|
||
- edits after the first test, total edits, and files changed;
|
||
- final test success, latency, and token usage.
|
||
|
||
Time to first edit is not a quality score. Interpret it together with
|
||
first-patch acceptance, rework, final success, and total cost.
|
||
|
||
## Install and run
|
||
|
||
From the repository root:
|
||
|
||
```bash
|
||
uv sync --locked --extra ch6
|
||
export OPENROUTER_API_KEY=...
|
||
uv run python chapter7/model-action-threshold/experiment.py \
|
||
--models openai/gpt-5.6-sol anthropic/claude-sonnet-5 \
|
||
--trials 3 \
|
||
--policy neutral \
|
||
--output chapter7/model-action-threshold/results/my-run
|
||
```
|
||
|
||
The runner alternates model order between trials and checkpoints the campaign
|
||
after every cell. Re-running the same command and output directory resumes
|
||
only the missing model × task × trial cells. `config.json` hashes the system prompt and tool schema;
|
||
`observations.jsonl` retains every trajectory; `summary.json` aggregates the
|
||
metrics; and `manifest.json` hashes those three artifacts.
|
||
|
||
Run the optional harness ablation separately:
|
||
|
||
```bash
|
||
uv run python chapter7/model-action-threshold/experiment.py \
|
||
--models openai/gpt-5.6-sol anthropic/claude-sonnet-5 \
|
||
--trials 3 --policy explore-first \
|
||
--output chapter7/model-action-threshold/results/explore-first
|
||
```
|
||
|
||
Do not merge neutral and explore-first observations into one model comparison.
|
||
The first run estimates the model effect under a neutral harness; comparing
|
||
the two campaigns estimates how much an explicit harness instruction modifies
|
||
that behavior.
|
||
|
||
## Validate the implementation
|
||
|
||
The offline tests verify path confinement, event-boundary accounting, rework
|
||
measurement, aggregation, and that every fixture starts in the intended
|
||
failing state:
|
||
|
||
```bash
|
||
python -m unittest discover -s chapter7/model-action-threshold/tests -v
|
||
```
|
||
|
||
The saved validation campaign in `results/` is considered complete only when
|
||
its manifest contains every requested model × task × trial observation and no
|
||
API errors. Model task failures remain valid experimental outcomes and are not
|
||
silently discarded.
|