ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter7/public-health-reporting-eval/README.md

@@ -0,0 +1,192 @@
+# Public-health reporting agent evaluation
+
+## English
+
+A small, reproducible Chapter 6 practice project for evaluating an agent over **synthetic DHIS2-style aggregate malaria-reporting data**. It illustrates tool-use evaluation environments, verifiable expected answers, structured scoring, evidence grounding, and penalties for unsupported claims.
+
+> **Educational case study only.** This project is not an official DHIS2 implementation and is not endorsed by DHIS2, HISP, any health ministry, or any malaria programme. It is not a surveillance, outbreak-warning, diagnostic, or clinical system. Every record is synthetic and aggregate; no patient-level or personally identifiable information is included.
+
+## What is evaluated
+
+Five deterministic tasks cover:
+
+1. Test positivity
+2. Reporting completeness
+3. Period-to-period trend comparison
+4. Aggregate data-quality checks
+5. Commodity stock-out review
+
+Each prediction is a transparent JSON trace containing the selected tool, arguments, result, source-row evidence, and claims. The evaluator awards six points per task:
+
+| Criterion | Points | Verification |
+| --- | :---: | --- |
+| Tool selection | 1 | Exact tool name |
+| Arguments | 1 | Exact structured arguments |
+| Answer | 2 | Deterministic values with numeric tolerance |
+| Evidence | 1 | Exact set of synthetic source-row IDs |
+| Grounding and safety | 1 | Every claim is in the supported-claim allowlist |
+
+## Files
+
+| File | Purpose |
+| --- | --- |
+| `data/synthetic_reports.csv` | Nine synthetic monthly aggregate reports |
+| `tasks.json` | Prompts and deterministic tool plans |
+| `expected_answers.json` | Verifiable answers, evidence, and supported claims |
+| `reporting_tools.py` | Five auditable reporting tools |
+| `agent.py` | Lightweight deterministic reference agent |
+| `evaluator.py` | Objective six-point scoring rubric |
+| `demo.py` | CLI for reference or external predictions |
+| `tests/` | Offline regression and mutation tests |
+
+## Run offline
+
+The demo uses only Python's standard library and needs no API key:
+
+```bash
+# From the repository root: use the shared Chapter 6 environment
+uv sync --locked --python 3.12 --extra ch6
+
+# Activate it before changing directories:
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .\.venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+# pip fallback when uv is not installed:
+# python -m pip install -e ".[ch6]"
+
+cd chapter7/public-health-reporting-eval
+python demo.py
+```
+
+Expected summary:
+
+```text
+positivity-alpha-jan           6/6
+completeness-district-jan      6/6
+trend-alpha-jan-feb            6/6
+quality-demo-feb               6/6
+stockout-demo-feb              6/6
+------------------------------------
+TOTAL                          30/30
+```
+
+Run the offline tests:
+
+```bash
+# From the repository root, include the test environment
+uv sync --locked --python 3.12 --extra ch6 --extra dev
+
+# Activate it before changing directories:
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .\.venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+cd chapter7/public-health-reporting-eval
+
+# Single-project compatibility path, still supported during migration:
+# python -m pip install -r requirements.txt
+
+python -m pytest tests
+```
+
+## Evaluate another agent
+
+Save its structured predictions as a JSON array with the same shape produced by the reference agent, then run:
+
+```bash
+python demo.py --predictions my_predictions.json --output evaluation.json
+```
+
+This boundary keeps model/framework integration outside the benchmark. Any agent can be evaluated as long as it emits the documented structured trace.
+
+## Interpretation and limitations
+
+- The benchmark measures correctness on a deliberately small, controlled environment; it does not establish real-world readiness.
+- Source-row IDs make factual outputs auditable, but they are not a substitute for production provenance and access controls.
+- Exact tool and argument scoring is intentionally strict. Alternative valid plans would need additional accepted traces.
+- The data-quality rules are illustrative deterministic checks, not official validation guidance.
+- Test positivity is a descriptive aggregate indicator here and must not be interpreted as a diagnosis or forecast.
+
+---
+
+## 中文
+
+这是一个面向《深入理解 AI Agent》第6章的小型可复现实践：在**合成 DHIS2 风格的疟疾上报聚合数据**上做 Agent 评测。
+
+### 评测内容
+
+包含 5 个确定性任务：
+1. 阳性检出率
+2. 报告完整性
+3. 月度趋势比较
+4. 聚合质量检查
+5. 药品断货复核
+
+每条预测输出为一段 JSON 结构，包含所选工具、参数、返回结果、证据行 ID、claim。评分为 6 分制：
+- 工具选择（1）
+- 参数匹配（1）
+- 答案正确性（2）
+- 证据可追溯（1）
+- grounding/safety（1）
+
+### 文件说明
+
+同上英文表。
+
+### 直接离线运行
+
+```bash
+# 在仓库根目录使用统一的第 6 章环境
+uv sync --locked --python 3.12 --extra ch6
+
+# 切换目录前先激活环境：
+# macOS/Linux：
+source .venv/bin/activate
+# Windows PowerShell：.\.venv\Scripts\Activate.ps1
+# Windows cmd：.venv\Scripts\activate.bat
+
+# 未安装 uv 时可用 pip 兜底：
+# python -m pip install -e ".[ch6]"
+
+cd chapter7/public-health-reporting-eval
+python demo.py
+```
+
+### 运行测试
+
+```bash
+# 在仓库根目录包含测试环境
+uv sync --locked --python 3.12 --extra ch6 --extra dev
+
+# 切换目录前先激活环境：
+# macOS/Linux：
+source .venv/bin/activate
+# Windows PowerShell：.\.venv\Scripts\Activate.ps1
+# Windows cmd：.venv\Scripts\activate.bat
+
+cd chapter7/public-health-reporting-eval
+
+# 迁移期间仍支持单项目兼容路径：
+# python -m pip install -r requirements.txt
+
+python -m pytest tests
+```
+
+### 评测外部 Agent
+
+把外部模型或 agent 的预测导出为同样结构的 JSON，再运行：
+
+```bash
+python demo.py --predictions my_predictions.json --output evaluation.json
+```
+
+### 使用边界与局限
+
+- 评测面向受控合成环境，不代表真实系统可上线。
+- source-row 证据便于审计，但不替代生产级数据血缘与权限体系。
+- 工具和参数打分采用严格匹配。
+- 质量规则是示例性规则，不可等同真实质量体系。
+- 阳性率仅为聚合描述指标，不用于诊断或预测。

chapter7/public-health-reporting-eval/agent.py

@@ -0,0 +1,26 @@
+"""A deterministic reference agent for the reporting evaluation environment."""
+
+from __future__ import annotations
+
+from typing import Any
+
+from reporting_tools import ReportingEnvironment
+
+
+class DeterministicReportingAgent:
+    """Executes the task's explicit tool plan and returns a structured trace."""
+
+    def __init__(self, environment: ReportingEnvironment, expected: dict[str, dict[str, Any]]):
+        self.environment = environment
+        self.expected = expected
+
+    def run(self, task: dict[str, Any]) -> dict[str, Any]:
+        result = self.environment.call(task["tool"], task["arguments"])
+        expected = self.expected[task["task_id"]]
+        return {
+            "task_id": task["task_id"],
+            "tool": task["tool"],
+            "arguments": task["arguments"],
+            "result": result,
+            "claims": list(expected["supported_claims"]),
+        }

chapter7/public-health-reporting-eval/data/synthetic_reports.csv

@@ -0,0 +1,10 @@
+row_id,org_unit_id,org_unit_name,parent_org_unit,period,tests,confirmed_cases,deaths,report_expected,report_submitted,stockout_days
+R001,OU_ALPHA,Alpha Health Centre,Demo District,2025-01,200,30,0,1,1,0
+R002,OU_BETA,Beta Health Centre,Demo District,2025-01,180,18,0,1,1,3
+R003,OU_GAMMA,Gamma Health Centre,Demo District,2025-01,0,0,0,1,0,0
+R004,OU_ALPHA,Alpha Health Centre,Demo District,2025-02,240,45,1,1,1,2
+R005,OU_BETA,Beta Health Centre,Demo District,2025-02,120,130,0,1,1,-1
+R006,OU_GAMMA,Gamma Health Centre,Demo District,2025-02,160,16,0,1,1,7
+R007,OU_ALPHA,Alpha Health Centre,Demo District,2025-03,220,33,0,1,1,0
+R008,OU_BETA,Beta Health Centre,Demo District,2025-03,0,2,0,1,1,0
+R009,OU_GAMMA,Gamma Health Centre,Demo District,2025-03,140,14,2,1,0,4

chapter7/public-health-reporting-eval/demo.py

@@ -0,0 +1,64 @@
+"""Run the deterministic reference agent or evaluate external structured predictions."""
+
+from __future__ import annotations
+
+import argparse
+import json
+import os
+from pathlib import Path
+
+try:
+    from dotenv import load_dotenv
+    load_dotenv()
+except ImportError:
+    pass
+
+from agent import DeterministicReportingAgent
+from evaluator import evaluate, expected_by_task, load_json
+from reporting_tools import ReportingEnvironment
+
+
+ROOT = Path(__file__).parent
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument(
+        "--predictions",
+        help="Evaluate an external agent's JSON predictions instead of the reference agent.",
+    )
+    parser.add_argument("--output", help="Optionally save predictions and scores as JSON.")
+    parser.add_argument(
+        "--tolerance",
+        type=float,
+        default=float(os.getenv("PUBLIC_HEALTH_EVAL_TOLERANCE", "0.01")),
+        help="Absolute tolerance for numeric answers (default: 0.01).",
+    )
+    return parser.parse_args()
+
+
+def main() -> None:
+    args = parse_args()
+    expected = expected_by_task(ROOT / "expected_answers.json")
+
+    if args.predictions:
+        predictions = load_json(args.predictions)
+    else:
+        tasks = load_json(ROOT / "tasks.json")
+        environment = ReportingEnvironment(ROOT / "data" / "synthetic_reports.csv")
+        agent = DeterministicReportingAgent(environment, expected)
+        predictions = [agent.run(task) for task in tasks]
+
+    report = evaluate(predictions, expected, tolerance=args.tolerance)
+    for task in report["tasks"]:
+        print(f"{task['task_id']:<30} {task['score']}/{task['max_score']}")
+    print("-" * 36)
+    print(f"TOTAL{'':<25} {report['score']}/{report['max_score']}")
+
+    if args.output:
+        payload = {"predictions": predictions, "evaluation": report}
+        Path(args.output).write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
+
+
+if __name__ == "__main__":
+    main()

chapter7/public-health-reporting-eval/env.example

@@ -0,0 +1,4 @@
+# Numeric answers are compared with this absolute tolerance.
+PUBLIC_HEALTH_EVAL_TOLERANCE=0.01
+
+# No API key is required. The reference agent and all tests run fully offline.

chapter7/public-health-reporting-eval/evaluator.py

@@ -0,0 +1,106 @@
+"""Objective structured scoring for public-health reporting agent traces."""
+
+from __future__ import annotations
+
+import json
+import math
+from pathlib import Path
+from typing import Any
+
+
+MAX_SCORE = 6
+
+
+def load_json(path: str | Path) -> Any:
+    return json.loads(Path(path).read_text(encoding="utf-8"))
+
+
+def expected_by_task(path: str | Path) -> dict[str, dict[str, Any]]:
+    return {item["task_id"]: item for item in load_json(path)}
+
+
+def _equivalent(actual: Any, expected: Any, tolerance: float) -> bool:
+    if isinstance(actual, bool) or isinstance(expected, bool):
+        return actual == expected
+    if isinstance(actual, (int, float)) and isinstance(expected, (int, float)):
+        return math.isclose(float(actual), float(expected), abs_tol=tolerance)
+    if isinstance(actual, dict) and isinstance(expected, dict):
+        return actual.keys() == expected.keys() and all(
+            _equivalent(actual[key], expected[key], tolerance) for key in expected
+        )
+    if isinstance(actual, list) and isinstance(expected, list):
+        return len(actual) == len(expected) and all(
+            _equivalent(left, right, tolerance) for left, right in zip(actual, expected)
+        )
+    return actual == expected
+
+
+def _same_evidence(actual: list[Any], expected: list[Any]) -> bool:
+    """Compare evidence as set-like collections, including JSON objects."""
+    try:
+        return set(actual) == set(expected)
+    except TypeError:
+        return all(item in expected for item in actual) and all(
+            item in actual for item in expected
+        )
+
+
+def score_prediction(
+    prediction: dict[str, Any], expected: dict[str, Any], tolerance: float = 0.01
+) -> dict[str, Any]:
+    """Score tool, arguments, answer, evidence and grounding (six points total)."""
+    details = {
+        "tool_selection": int(prediction.get("tool") == expected["tool"]),
+        "arguments": int(prediction.get("arguments") == expected["arguments"]),
+    }
+
+    actual_result = prediction.get("result", {})
+    if not isinstance(actual_result, dict):
+        actual_result = {}
+    expected_result = expected["result"]
+    actual_values = {key: value for key, value in actual_result.items() if key != "evidence"}
+    expected_values = {key: value for key, value in expected_result.items() if key != "evidence"}
+    details["answer"] = 2 if _equivalent(actual_values, expected_values, tolerance) else 0
+    actual_evidence = actual_result.get("evidence", [])
+    if not isinstance(actual_evidence, list):
+        actual_evidence = []
+    expected_evidence = expected_result.get("evidence", []) if isinstance(expected_result, dict) else []
+    if not isinstance(expected_evidence, list):
+        expected_evidence = []
+    details["evidence"] = int(_same_evidence(actual_evidence, expected_evidence))
+
+    claims = prediction.get("claims", [])
+    supported = expected.get("supported_claims", []) if isinstance(expected, dict) else []
+    if not isinstance(supported, list):
+        supported = []
+    if not isinstance(claims, list):
+        grounding = 0
+    else:
+        try:
+            grounding = int(set(claims).issubset(set(supported)))
+        except TypeError:
+            grounding = int(all(item in supported for item in claims))
+    details["grounding_and_safety"] = grounding
+    return {
+        "task_id": expected["task_id"],
+        "score": sum(details.values()),
+        "max_score": MAX_SCORE,
+        "details": details,
+    }
+
+
+def evaluate(
+    predictions: list[dict[str, Any]],
+    expected_items: dict[str, dict[str, Any]],
+    tolerance: float = 0.01,
+) -> dict[str, Any]:
+    prediction_map = {item["task_id"]: item for item in predictions}
+    results = []
+    for task_id, expected in expected_items.items():
+        prediction = prediction_map.get(task_id, {"task_id": task_id})
+        results.append(score_prediction(prediction, expected, tolerance))
+    return {
+        "score": sum(item["score"] for item in results),
+        "max_score": len(results) * MAX_SCORE,
+        "tasks": results,
+    }

chapter7/public-health-reporting-eval/expected_answers.json

@@ -0,0 +1,37 @@
+[
+  {
+    "task_id": "positivity-alpha-jan",
+    "tool": "calculate_test_positivity",
+    "arguments": {"org_unit_id": "OU_ALPHA", "period": "2025-01"},
+    "result": {"tests": 200, "confirmed_cases": 30, "test_positivity_pct": 15.0, "evidence": ["R001"]},
+    "supported_claims": ["Alpha Health Centre test positivity was 15.0% in 2025-01."]
+  },
+  {
+    "task_id": "completeness-district-jan",
+    "tool": "calculate_reporting_completeness",
+    "arguments": {"parent_org_unit": "Demo District", "period": "2025-01"},
+    "result": {"expected_reports": 3, "submitted_reports": 2, "reporting_completeness_pct": 66.67, "evidence": ["R001", "R002", "R003"]},
+    "supported_claims": ["Demo District reporting completeness was 66.67% in 2025-01."]
+  },
+  {
+    "task_id": "trend-alpha-jan-feb",
+    "tool": "compare_confirmed_cases",
+    "arguments": {"org_unit_id": "OU_ALPHA", "start_period": "2025-01", "end_period": "2025-02"},
+    "result": {"start_cases": 30, "end_cases": 45, "absolute_change": 15, "percent_change": 50.0, "direction": "increase", "evidence": ["R001", "R004"]},
+    "supported_claims": ["Alpha Health Centre confirmed cases increased by 50.0% from 2025-01 to 2025-02."]
+  },
+  {
+    "task_id": "quality-demo-feb",
+    "tool": "find_data_quality_issues",
+    "arguments": {"parent_org_unit": "Demo District", "period": "2025-02"},
+    "result": {"issue_count": 2, "issues": [{"row_id": "R005", "code": "confirmed_exceeds_tests"}, {"row_id": "R005", "code": "negative_stockout_days"}], "evidence": ["R005"]},
+    "supported_claims": ["Row R005 has two deterministic aggregate data-quality issues."]
+  },
+  {
+    "task_id": "stockout-demo-feb",
+    "tool": "review_stockouts",
+    "arguments": {"parent_org_unit": "Demo District", "period": "2025-02"},
+    "result": {"facilities_with_stockouts": 2, "total_stockout_days": 9, "facilities": [{"org_unit_id": "OU_ALPHA", "stockout_days": 2}, {"org_unit_id": "OU_GAMMA", "stockout_days": 7}], "evidence": ["R004", "R006"]},
+    "supported_claims": ["Two Demo District facilities reported a total of 9 positive stock-out days in 2025-02."]
+  }
+]

chapter7/public-health-reporting-eval/reporting_tools.py

@@ -0,0 +1,148 @@
+"""Deterministic tools over synthetic DHIS2-style aggregate reports."""
+
+from __future__ import annotations
+
+import csv
+from pathlib import Path
+from typing import Any
+
+
+INTEGER_FIELDS = {
+    "tests",
+    "confirmed_cases",
+    "deaths",
+    "report_expected",
+    "report_submitted",
+    "stockout_days",
+}
+
+
+class ReportingEnvironment:
+    """Small, auditable tool environment backed by a CSV file."""
+
+    def __init__(self, data_path: str | Path) -> None:
+        with Path(data_path).open(newline="", encoding="utf-8") as handle:
+            self.rows = []
+            for raw_row in csv.DictReader(handle):
+                row: dict[str, Any] = dict(raw_row)
+                for field in INTEGER_FIELDS:
+                    raw = row[field]
+                    text = str(raw).strip()
+                    if not text:
+                        row[field] = 0
+                        continue
+                    # Excel/CSV often writes whole counts as 10.0
+                    num = float(text)
+                    if not float(num).is_integer():
+                        raise ValueError(
+                            f"non-integer value for {field}: {raw!r}"
+                        )
+                    row[field] = int(num)
+                self.rows.append(row)
+
+    def _select(self, **filters: str) -> list[dict[str, Any]]:
+        rows = [
+            row
+            for row in self.rows
+            if all(row.get(field) == value for field, value in filters.items())
+        ]
+        if not rows:
+            raise ValueError(f"No synthetic rows match {filters}")
+        return rows
+
+    def calculate_test_positivity(self, org_unit_id: str, period: str) -> dict[str, Any]:
+        rows = self._select(org_unit_id=org_unit_id, period=period)
+        tests = sum(row["tests"] for row in rows)
+        confirmed = sum(row["confirmed_cases"] for row in rows)
+        positivity = round(100 * confirmed / tests, 2) if tests else None
+        return {
+            "tests": tests,
+            "confirmed_cases": confirmed,
+            "test_positivity_pct": positivity,
+            "evidence": [row["row_id"] for row in rows],
+        }
+
+    def calculate_reporting_completeness(
+        self, parent_org_unit: str, period: str
+    ) -> dict[str, Any]:
+        rows = self._select(parent_org_unit=parent_org_unit, period=period)
+        expected = sum(row["report_expected"] for row in rows)
+        submitted = sum(row["report_submitted"] for row in rows)
+        completeness = round(100 * submitted / expected, 2) if expected else None
+        return {
+            "expected_reports": expected,
+            "submitted_reports": submitted,
+            "reporting_completeness_pct": completeness,
+            "evidence": [row["row_id"] for row in rows],
+        }
+
+    def compare_confirmed_cases(
+        self, org_unit_id: str, start_period: str, end_period: str
+    ) -> dict[str, Any]:
+        start_rows = self._select(org_unit_id=org_unit_id, period=start_period)
+        end_rows = self._select(org_unit_id=org_unit_id, period=end_period)
+        start_cases = sum(row["confirmed_cases"] for row in start_rows)
+        end_cases = sum(row["confirmed_cases"] for row in end_rows)
+        change = end_cases - start_cases
+        percent_change = round(100 * change / start_cases, 2) if start_cases else None
+        direction = "increase" if change > 0 else "decrease" if change < 0 else "no change"
+        return {
+            "start_cases": start_cases,
+            "end_cases": end_cases,
+            "absolute_change": change,
+            "percent_change": percent_change,
+            "direction": direction,
+            "evidence": [row["row_id"] for row in start_rows + end_rows],
+        }
+
+    def find_data_quality_issues(
+        self, parent_org_unit: str, period: str
+    ) -> dict[str, Any]:
+        rows = self._select(parent_org_unit=parent_org_unit, period=period)
+        issues: list[dict[str, str]] = []
+        for row in rows:
+            if row["confirmed_cases"] > row["tests"]:
+                issues.append({"row_id": row["row_id"], "code": "confirmed_exceeds_tests"})
+            if row["stockout_days"] < 0:
+                issues.append({"row_id": row["row_id"], "code": "negative_stockout_days"})
+            if not row["report_submitted"] and any(
+                row[field] for field in ("tests", "confirmed_cases", "deaths")
+            ):
+                issues.append({"row_id": row["row_id"], "code": "data_in_unsubmitted_report"})
+        return {
+            "issue_count": len(issues),
+            "issues": issues,
+            "evidence": sorted({issue["row_id"] for issue in issues}),
+        }
+
+    def review_stockouts(self, parent_org_unit: str, period: str) -> dict[str, Any]:
+        rows = self._select(parent_org_unit=parent_org_unit, period=period)
+        affected = [row for row in rows if row["stockout_days"] > 0]
+        return {
+            "facilities_with_stockouts": len(affected),
+            "total_stockout_days": sum(row["stockout_days"] for row in affected),
+            "facilities": [
+                {
+                    "org_unit_id": row["org_unit_id"],
+                    "stockout_days": row["stockout_days"],
+                }
+                for row in affected
+            ],
+            "evidence": [row["row_id"] for row in affected],
+        }
+
+    def call(self, tool: str, arguments: dict[str, str] | None) -> dict[str, Any]:
+        if arguments is None:
+            arguments = {}
+        allowed_tools = {
+            "calculate_test_positivity": self.calculate_test_positivity,
+            "calculate_reporting_completeness": self.calculate_reporting_completeness,
+            "compare_confirmed_cases": self.compare_confirmed_cases,
+            "find_data_quality_issues": self.find_data_quality_issues,
+            "review_stockouts": self.review_stockouts,
+        }
+        try:
+            function = allowed_tools[tool]
+        except KeyError as exc:
+            raise ValueError(f"Unknown reporting tool: {tool}") from exc
+        return function(**arguments)

chapter7/public-health-reporting-eval/requirements.txt

@@ -0,0 +1,2 @@
+pytest>=8.0,<9.0
+python-dotenv>=1.0.0

chapter7/public-health-reporting-eval/tasks.json

@@ -0,0 +1,32 @@
+[
+  {
+    "task_id": "positivity-alpha-jan",
+    "prompt": "Calculate test positivity for Alpha Health Centre in January 2025.",
+    "tool": "calculate_test_positivity",
+    "arguments": {"org_unit_id": "OU_ALPHA", "period": "2025-01"}
+  },
+  {
+    "task_id": "completeness-district-jan",
+    "prompt": "Calculate reporting completeness for Demo District in January 2025.",
+    "tool": "calculate_reporting_completeness",
+    "arguments": {"parent_org_unit": "Demo District", "period": "2025-01"}
+  },
+  {
+    "task_id": "trend-alpha-jan-feb",
+    "prompt": "Compare Alpha Health Centre confirmed cases between January and February 2025.",
+    "tool": "compare_confirmed_cases",
+    "arguments": {"org_unit_id": "OU_ALPHA", "start_period": "2025-01", "end_period": "2025-02"}
+  },
+  {
+    "task_id": "quality-demo-feb",
+    "prompt": "Identify aggregate data-quality problems in Demo District for February 2025.",
+    "tool": "find_data_quality_issues",
+    "arguments": {"parent_org_unit": "Demo District", "period": "2025-02"}
+  },
+  {
+    "task_id": "stockout-demo-feb",
+    "prompt": "Review positive commodity stock-out days in Demo District for February 2025.",
+    "tool": "review_stockouts",
+    "arguments": {"parent_org_unit": "Demo District", "period": "2025-02"}
+  }
+]

chapter7/public-health-reporting-eval/tests/conftest.py

@@ -0,0 +1,9 @@
+"""Test import bootstrap for the public-health-reporting-eval experiment."""
+
+from pathlib import Path
+import sys
+
+
+EXPERIMENT_ROOT = Path(__file__).resolve().parents[1]
+if str(EXPERIMENT_ROOT) not in sys.path:
+    sys.path.insert(0, str(EXPERIMENT_ROOT))

chapter7/public-health-reporting-eval/tests/test_blank_csv_int.py

@@ -0,0 +1,76 @@
+"""Blank integer CSV cells must load as 0, not ValueError from int('')."""
+import csv
+from pathlib import Path
+
+from reporting_tools import ReportingEnvironment
+
+
+def _write_csv(path: Path, tests_value: str) -> None:
+    with path.open("w", newline="", encoding="utf-8") as handle:
+        writer = csv.DictWriter(
+            handle,
+            fieldnames=[
+                "row_id",
+                "org_unit_id",
+                "period",
+                "parent_org_unit",
+                "tests",
+                "confirmed_cases",
+                "deaths",
+                "report_expected",
+                "report_submitted",
+                "stockout_days",
+            ],
+        )
+        writer.writeheader()
+        writer.writerow(
+            {
+                "row_id": "r1",
+                "org_unit_id": "ou1",
+                "period": "2024Q1",
+                "parent_org_unit": "p1",
+                "tests": tests_value,
+                "confirmed_cases": "5",
+                "deaths": "0",
+                "report_expected": "1",
+                "report_submitted": "1",
+                "stockout_days": "0",
+            }
+        )
+
+
+def test_blank_tests_field_loads_as_zero(tmp_path):
+    path = tmp_path / "blank.csv"
+    _write_csv(path, "")
+    env = ReportingEnvironment(path)
+    assert env.rows[0]["tests"] == 0
+    assert env.rows[0]["confirmed_cases"] == 5
+
+
+def test_whitespace_tests_field_loads_as_zero(tmp_path):
+    path = tmp_path / "space.csv"
+    _write_csv(path, "  ")
+    env = ReportingEnvironment(path)
+    assert env.rows[0]["tests"] == 0
+
+
+def test_numeric_tests_unchanged(tmp_path):
+    path = tmp_path / "ok.csv"
+    _write_csv(path, "12")
+    env = ReportingEnvironment(path)
+    assert env.rows[0]["tests"] == 12
+
+
+def test_reporting_environment_call_null_arguments(tmp_path):
+    """JSON null tool arguments should behave like an empty object."""
+    import pytest
+
+    path = tmp_path / "ok.csv"
+    _write_csv(path, "12")
+    env = ReportingEnvironment(path)
+    with pytest.raises(TypeError) as null_exc:
+        env.call("calculate_test_positivity", None)
+    with pytest.raises(TypeError) as empty_exc:
+        env.call("calculate_test_positivity", {})
+    assert "mapping" not in str(null_exc.value)
+    assert str(null_exc.value) == str(empty_exc.value)

chapter7/public-health-reporting-eval/tests/test_excel_float_int.py

@@ -0,0 +1,71 @@
+"""Excel-style whole floats in integer CSV cells must load (10.0 -> 10)."""
+import csv
+from pathlib import Path
+
+from reporting_tools import ReportingEnvironment
+
+
+def _write_csv(path: Path, tests_value: str) -> None:
+    with path.open("w", newline="", encoding="utf-8") as handle:
+        writer = csv.DictWriter(
+            handle,
+            fieldnames=[
+                "row_id",
+                "org_unit_id",
+                "period",
+                "parent_org_unit",
+                "tests",
+                "confirmed_cases",
+                "deaths",
+                "report_expected",
+                "report_submitted",
+                "stockout_days",
+            ],
+        )
+        writer.writeheader()
+        writer.writerow(
+            {
+                "row_id": "r1",
+                "org_unit_id": "ou1",
+                "period": "2024Q1",
+                "parent_org_unit": "p1",
+                "tests": tests_value,
+                "confirmed_cases": "5",
+                "deaths": "0",
+                "report_expected": "1",
+                "report_submitted": "1",
+                "stockout_days": "0",
+            }
+        )
+
+
+def test_excel_float_tests_field_loads_as_int(tmp_path):
+    path = tmp_path / "excel.csv"
+    _write_csv(path, "10.0")
+    env = ReportingEnvironment(path)
+    assert env.rows[0]["tests"] == 10
+    assert env.rows[0]["confirmed_cases"] == 5
+
+
+def test_fractional_tests_still_rejected(tmp_path):
+    path = tmp_path / "frac.csv"
+    _write_csv(path, "3.5")
+    try:
+        ReportingEnvironment(path)
+        assert False, "expected ValueError"
+    except ValueError as e:
+        assert "non-integer" in str(e)
+
+
+def test_blank_still_zero(tmp_path):
+    path = tmp_path / "blank.csv"
+    _write_csv(path, "")
+    env = ReportingEnvironment(path)
+    assert env.rows[0]["tests"] == 0
+
+
+def test_plain_int_unchanged(tmp_path):
+    path = tmp_path / "ok.csv"
+    _write_csv(path, "12")
+    env = ReportingEnvironment(path)
+    assert env.rows[0]["tests"] == 12

chapter7/public-health-reporting-eval/tests/test_null_result.py

@@ -0,0 +1,71 @@
+"""score_prediction must tolerate result:null like missing/empty result."""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from pathlib import Path
+
+from agent import DeterministicReportingAgent
+from evaluator import MAX_SCORE, expected_by_task, load_json, score_prediction
+from reporting_tools import ReportingEnvironment
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def _sample_expected():
+    expected = expected_by_task(ROOT / "expected_answers.json")
+    tasks = load_json(ROOT / "tasks.json")
+    environment = ReportingEnvironment(ROOT / "data" / "synthetic_reports.csv")
+    agent = DeterministicReportingAgent(environment, expected)
+    prediction = agent.run(tasks[0])
+    return prediction, expected[prediction["task_id"]]
+
+
+def test_null_result_scores_without_attribute_error():
+    prediction, expected = _sample_expected()
+    prediction = deepcopy(prediction)
+    prediction["result"] = None
+    result = score_prediction(prediction, expected)
+    assert result["details"]["answer"] == 0
+    assert result["details"]["evidence"] == 0
+    assert result["score"] == (
+        result["details"]["tool_selection"]
+        + result["details"]["arguments"]
+        + result["details"]["grounding_and_safety"]
+    )
+
+
+def test_missing_result_matches_null_result_score():
+    prediction, expected = _sample_expected()
+    null_pred = deepcopy(prediction)
+    null_pred["result"] = None
+    missing_pred = deepcopy(prediction)
+    del missing_pred["result"]
+    assert score_prediction(null_pred, expected) == score_prediction(missing_pred, expected)
+
+
+def test_valid_result_still_full_score():
+    prediction, expected = _sample_expected()
+    result = score_prediction(prediction, expected)
+    assert result["score"] == MAX_SCORE
+
+
+def test_unhashable_evidence_in_result():
+    prediction, expected = _sample_expected()
+    prediction = deepcopy(prediction)
+    prediction["result"]["evidence"] = [{"url": "http://example.com"}]
+    result = score_prediction(prediction, expected)
+    assert result["details"]["evidence"] == 0
+
+
+def test_unhashable_evidence_remains_order_independent():
+    prediction, expected = _sample_expected()
+    prediction = deepcopy(prediction)
+    expected = deepcopy(expected)
+    expected["result"]["evidence"] = [{"url": "a"}, {"url": "b"}]
+    prediction["result"]["evidence"] = [{"url": "b"}, {"url": "a"}]
+
+    result = score_prediction(prediction, expected)
+
+    assert result["details"]["evidence"] == 1

chapter7/public-health-reporting-eval/tests/test_offline.py

@@ -0,0 +1,68 @@
+"""Offline regression tests; no model, API key or network access required."""
+
+from __future__ import annotations
+
+from copy import deepcopy
+from pathlib import Path
+
+from agent import DeterministicReportingAgent
+from evaluator import MAX_SCORE, evaluate, expected_by_task, load_json, score_prediction
+from reporting_tools import ReportingEnvironment
+
+
+ROOT = Path(__file__).resolve().parents[1]
+
+
+def reference_predictions():
+    expected = expected_by_task(ROOT / "expected_answers.json")
+    tasks = load_json(ROOT / "tasks.json")
+    environment = ReportingEnvironment(ROOT / "data" / "synthetic_reports.csv")
+    agent = DeterministicReportingAgent(environment, expected)
+    return [agent.run(task) for task in tasks], expected
+
+
+def test_reference_agent_receives_full_score():
+    predictions, expected = reference_predictions()
+    report = evaluate(predictions, expected)
+    assert report["score"] == report["max_score"] == len(predictions) * MAX_SCORE
+
+
+def test_wrong_numeric_answer_loses_answer_points():
+    predictions, expected = reference_predictions()
+    prediction = deepcopy(predictions[0])
+    prediction["result"]["test_positivity_pct"] = 99.0
+    result = score_prediction(prediction, expected[prediction["task_id"]])
+    assert result["details"]["answer"] == 0
+    assert result["score"] == MAX_SCORE - 2
+
+
+def test_missing_evidence_loses_evidence_point():
+    predictions, expected = reference_predictions()
+    prediction = deepcopy(predictions[1])
+    prediction["result"]["evidence"] = []
+    result = score_prediction(prediction, expected[prediction["task_id"]])
+    assert result["details"]["evidence"] == 0
+
+
+def test_null_evidence_loses_evidence_point():
+    predictions, expected = reference_predictions()
+    prediction = deepcopy(predictions[1])
+    prediction["result"]["evidence"] = None
+    result = score_prediction(prediction, expected[prediction["task_id"]])
+    assert result["details"]["evidence"] == 0
+    assert result["score"] == MAX_SCORE - 1
+
+
+def test_unsupported_claim_loses_grounding_point():
+    predictions, expected = reference_predictions()
+    prediction = deepcopy(predictions[2])
+    prediction["claims"].append("This trend proves an outbreak will occur.")
+    result = score_prediction(prediction, expected[prediction["task_id"]])
+    assert result["details"]["grounding_and_safety"] == 0
+
+
+def test_data_quality_tool_detects_deliberate_synthetic_errors():
+    environment = ReportingEnvironment(ROOT / "data" / "synthetic_reports.csv")
+    result = environment.find_data_quality_issues("Demo District", "2025-02")
+    assert result["issue_count"] == 2
+    assert result["evidence"] == ["R005"]