ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,4 @@
|
||||
.env
|
||||
__pycache__/
|
||||
*.pyc
|
||||
last_run.json
|
||||
@@ -0,0 +1,416 @@
|
||||
# Experiment 5-2: Code Tools for Logic / 实验 5-2：用代码生成工具提升逻辑思考能力
|
||||
|
||||
> Companion lab for *AI Agents in Depth*, Chapter 5 — Knights & Knaves as CSP with `python-constraint`; pure reasoning vs code-assisted vs offline solver.
|
||||
> 《深入理解 AI Agent》第 5 章配套：骑士与无赖谜题转 CSP，对比纯思考 / 代码辅助 / 离线约束求解。
|
||||
|
||||
← [Chapter 5 index / 返回第 5 章目录](../README.md)
|
||||
|
||||
## Formal manuscript result (canonical)
|
||||
|
||||
The canonical run uses the pinned revision of
|
||||
`K-and-K/perturbed-knights-and-knaves`, stratified across six perturbations and
|
||||
2–8 people (84 paired tasks). Every code trajectory invoked
|
||||
`python-constraint`, but observed accuracy was 39.3% for code assistance versus
|
||||
75.0% for pure reasoning (p=2.27e-7 in the opposite direction). The campaign is
|
||||
complete; the manuscript's >90% and significant-improvement hypothesis was not
|
||||
observed. Evidence: [`validation/real_ark_doubao_flash_hf84_20260730.json`](validation/real_ark_doubao_flash_hf84_20260730.json).
|
||||
|
||||
正式活动固定 K&K 数据集版本，按六类扰动与 2–8 人规模分层抽取 84 道配对题。代码臂
|
||||
每题都调用了 `python-constraint`，但实测准确率为 39.3%，低于纯思考的 75.0%
|
||||
（p=2.27e-7，方向与预期相反）。因此下文离线求解器 100% 的表格只能证明确定性 CSP
|
||||
机制正确，不能替代真实 LLM 对照或正文的 >90% 验收结论。
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
### Overview
|
||||
|
||||
This lab evaluates whether an Agent can use **constraint-solving** code to support logical thinking. The same LLM gets a Code Interpreter preloaded with `python-constraint`, and turns Knights & Knaves (K&K) puzzles into formal **constraint satisfaction problems (CSP)**—variables (each islander is knight or knave), constraints (“knights tell truth, knaves lie”), then a solver search.
|
||||
|
||||
On a set of 12 K&K puzzles (2–5 people, each with a unique truth assignment), three modes are compared:
|
||||
|
||||
- **Pure thinking (`pure`)**: natural-language chain-of-thought only; answer directly.
|
||||
- **Code-assisted (`code`)**: use `run_python` to write a constraint model and call the solver, then answer from the result.
|
||||
- **Constraint solver (`solver`)**: **offline baseline**—solve structured statements with `python-constraint` only, no API/network. Deterministic; theoretically 100% correct; validates “translate puzzle → constraints → solve” (see real results below).
|
||||
|
||||
### Core idea: why code helps
|
||||
|
||||
The key modeling rule for K&K is one **biconditional (equivalence)** per resident X:
|
||||
|
||||
```
|
||||
X is knight (True) <=> X's statement is true
|
||||
```
|
||||
|
||||
i.e. `X == (semantic truth of that statement)`. Hand this to a deterministic solver that **enumerates** all Boolean assignments and logic cannot “slip”; pure thinking often fails on multi-person, counting (“exactly two knights”), or self-referential (“A and B are the same type”) puzzles when propagating truth values by hand.
|
||||
|
||||
### Files
|
||||
|
||||
| File | Role |
|
||||
| --- | --- |
|
||||
| `demo.py` | Main: pure / code / solver comparison; accuracy table |
|
||||
| `csp_solver.py` | Offline CSP solver: structured statement DSL + `python-constraint` (shared by demo solver mode and `build_puzzles` checks) |
|
||||
| `sandbox.py` | Minimal Code Interpreter: subprocess sandbox for model-generated Python (python-constraint preinstalled) |
|
||||
| `puzzles.json` | 12 puzzles: stems + structured statements + unique solutions (LLM sees stems only) |
|
||||
| `build_puzzles.py` | Generate/validate puzzles: solve with `python-constraint`, assert unique solution; export curated or random sets |
|
||||
| `requirements.txt` | Dependencies (openai + python-constraint) |
|
||||
| `env.example` | Env var sample |
|
||||
| `last_run.json` | Full per-problem record after each run (including model-generated code) for review |
|
||||
|
||||
### Quick start
|
||||
|
||||
```bash
|
||||
# From the repository root: use the shared Chapter 5 environment
|
||||
uv sync --locked --python 3.12 --extra ch5
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch5]"
|
||||
|
||||
cd chapter5/code-for-logic
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
#### 1) Offline solver baseline (no API key; recommended first)
|
||||
|
||||
```bash
|
||||
python demo.py --mode solver # offline solve all 12 with python-constraint
|
||||
python demo.py --mode solver --min-people 4 # only puzzles with >=4 people
|
||||
```
|
||||
|
||||
Fully offline and deterministic; demonstrates “puzzle → constraints → solve” at 100% accuracy.
|
||||
|
||||
#### 2) LLM comparison (needs `OPENAI_API_KEY` or `OPENROUTER_API_KEY`)
|
||||
|
||||
```bash
|
||||
cp env.example .env # then edit .env with OPENAI_API_KEY
|
||||
# or: export OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
python demo.py # default both: pure vs code, all 12
|
||||
python demo.py --mode pure # pure baseline only
|
||||
python demo.py --limit 4 # first 4 only (cheap smoke)
|
||||
python demo.py --max-people 3 # only puzzles with <=3 people
|
||||
python demo.py --model gpt-4o-mini # model (default gpt-4o-mini)
|
||||
python demo.py --puzzles my.json --output run.json # custom data / output path
|
||||
```
|
||||
|
||||
**OpenRouter fallback**: if `OPENAI_API_KEY` is unset but `OPENROUTER_API_KEY` is set, traffic goes through OpenRouter (`gpt-*` → `openai/*`). Default `gpt-4o-mini` works on direct OpenAI; OpenRouter is preferred when you switch `--model` to gpt-5.x models that need org verification and `OPENROUTER_API_KEY` is set.
|
||||
|
||||
Full flags: `python demo.py --help` (Chinese help text).
|
||||
|
||||
#### 3) Build / expand the puzzle set
|
||||
|
||||
```bash
|
||||
python build_puzzles.py # export built-in 12 curated puzzles (default)
|
||||
python build_puzzles.py --generate 20 --min-people 3 --max-people 5 --seed 7
|
||||
python build_puzzles.py --generate 20 --output my.json
|
||||
```
|
||||
|
||||
The random generator solves each candidate with `python-constraint` and keeps only unique-solution puzzles.
|
||||
|
||||
`sandbox.py` / `csp_solver.py` can also be run alone for self-tests:
|
||||
`python sandbox.py`, `python csp_solver.py` each solve a minimal puzzle with python-constraint.
|
||||
|
||||
### Real results (1): offline solver (`--mode solver`, no API)
|
||||
|
||||
Actual output of `python demo.py --mode solver` (12 curated puzzles, offline, deterministic):
|
||||
|
||||
```
|
||||
== 约束求解(solver，离线) ==
|
||||
[solver] kk01 (2人) ✓ 解数=1 预测={'A': 'knight', 'B': 'knave'}
|
||||
[solver] kk05 (3人) ✓ 解数=1 预测={'A': 'knave', 'B': 'knave', 'C': 'knight'}
|
||||
[solver] kk11 (5人) ✓ 解数=1 预测={'A': 'knight', 'B': 'knight', 'C': 'knave', 'D': 'knave', 'E': 'knight'}
|
||||
...（其余题略）
|
||||
------------------------------------------------------------
|
||||
准确率 100.0%
|
||||
============================================================
|
||||
约束求解 准确率: 100.0% (12/12)
|
||||
```
|
||||
|
||||
This path translates each structured statement into `python-constraint` and enumerates—12/12 correct. It proves determinism of “puzzle → constraints → solve”; if the LLM translates correctly, it gets the same 100%. Random puzzles from `build_puzzles.py --generate` also solve 100% and match the unique solutions recorded at generation time.
|
||||
|
||||
### Real results (2): LLM comparison (gpt-4o-mini, 12 puzzles)
|
||||
|
||||
```
|
||||
准确率对比表
|
||||
============================================================
|
||||
题号 人数 纯思考 代码辅助
|
||||
------------------------------------------------------------
|
||||
kk01 2 ✓ ✓
|
||||
kk02 2 ✓ ✓
|
||||
kk03 2 ✓ ✓
|
||||
kk04 3 ✓ ✓
|
||||
kk05 3 ✗ ✓
|
||||
kk06 3 ✗ ✓
|
||||
kk07 3 ✗ ✓
|
||||
kk08 4 ✗ ✓
|
||||
kk09 4 ✗ ✓
|
||||
kk10 4 ✓ ✓
|
||||
kk11 5 ✗ ✓
|
||||
kk12 5 ✓ ✓
|
||||
------------------------------------------------------------
|
||||
准确率 50.0% 100.0%
|
||||
============================================================
|
||||
纯思考 准确率: 50.0% (6/12)
|
||||
代码辅助 准确率: 100.0% (12/12)
|
||||
提升(代码辅助 - 纯思考): +50.0 个百分点
|
||||
```
|
||||
|
||||
> A weaker `gpt-4o-mini` is used on purpose: pure thinking got **6/12 (50%)**, with errors concentrated on ≥3 people and counting/self-reference (kk05–kk09, kk11)—exactly where mental truth propagation fails; code-assisted maps each sentence to biconditionals and lets `python-constraint` enumerate for **12/12** and **+50 percentage points**. Correctness no longer depends on the model’s own reasoning strength. There is some run-to-run noise on individual items, but “pure ≪ code-assisted” is stable.
|
||||
|
||||
> **Model ↔ harness tradeoff**: stronger models need thinner harnesses; weaker models need more (e.g. offload logic to code/solvers). With weak `gpt-4o-mini` the contrast is visible; with strong reasoners like `gpt-5.6-luna`, pure thinking can also full-solve and code gains can go to 0. Code-assisted (and offline solver) turn correctness into something **deterministic and model-strength-independent**.
|
||||
|
||||
#### Example constraint code (model-generated, kk11, 5 people + count)
|
||||
|
||||
Stem: A says “B is a knight”; B says “C is a knave”; C says “D is a knight”; D says “E is a knave”;
|
||||
E says “at least two of us five are knights”.
|
||||
|
||||
```python
|
||||
from constraint import Problem
|
||||
|
||||
p = Problem()
|
||||
for name in ['A', 'B', 'C', 'D', 'E']:
|
||||
p.addVariable(name, [True, False]) # True=knight (truth), False=knave (lie)
|
||||
|
||||
# Each sentence: X == (truth value of the claim)
|
||||
p.addConstraint(lambda a, b: a == (b == True), ['A', 'B']) # A:"B is knight"
|
||||
p.addConstraint(lambda b, c: b == (c == False), ['B', 'C']) # B:"C is knave"
|
||||
p.addConstraint(lambda c, d: c == (d == True), ['C', 'D']) # C:"D is knight"
|
||||
p.addConstraint(lambda d, e: d == (e == False), ['D', 'E']) # D:"E is knave"
|
||||
p.addConstraint(lambda a, b, c, d, e: e == ((a + b + c + d + e) >= 2),
|
||||
['A', 'B', 'C', 'D', 'E']) # E:"at least two knights"
|
||||
|
||||
for s in p.getSolutions():
|
||||
print({k: ('knight' if v else 'knave') for k, v in s.items()})
|
||||
# Output: {'A': 'knight', 'B': 'knight', 'C': 'knave', 'D': 'knave', 'E': 'knight'}
|
||||
```
|
||||
|
||||
The solver enumerates \(2^5=32\) assignments and returns the unique solution—the kind of chain pure thinking most often gets wrong.
|
||||
|
||||
### Notes
|
||||
|
||||
- **Cost**: default `gpt-4o-mini` (weaker model for contrast); 12 puzzles × two modes is cheap; override with `MODEL` / `--model`.
|
||||
- **API key**: `OPENAI_API_KEY` or `OPENROUTER_API_KEY` from env / `.env`; `MODEL` to switch models.
|
||||
- **Sandbox**: `sandbox.py` uses subprocess + timeout—teaching minimal sandbox; production should use containers/gVisor etc.
|
||||
- **Puzzle reliability**: `build_puzzles.py` solves each puzzle (curated or random) with `python-constraint` and asserts unique solution before writing; extend via `CURATED` or `--generate`.
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 概述
|
||||
|
||||
本实验评估 Agent 通过**约束求解**代码来辅助逻辑思考的能力：为同一个 LLM 配备一个预装
|
||||
`python-constraint` 的 Code Interpreter，让它把「骑士与无赖」(Knights & Knaves) 逻辑谜题
|
||||
转化为形式化的**约束满足问题(CSP)**——识别变量(每个岛民是骑士还是无赖)、定义约束
|
||||
(“骑士说真话、无赖说假话”)，再调用求解器搜索满足所有约束的解。
|
||||
|
||||
我们用一组 12 道 K&K 谜题(2~5 人，均带唯一真值解)对比三种模式：
|
||||
|
||||
- **纯思考(pure)**：LLM 只用自然语言链式推理，直接给答案；
|
||||
- **代码辅助(code)**：LLM 用 `run_python` 工具写约束模型并调求解器，再据结果作答；
|
||||
- **约束求解(solver)**：**离线基线**，直接用 `python-constraint` 求解结构化陈述，
|
||||
不需要任何 API/网络——它是确定性求解器路径本身，理论上 100% 正确，用来验证
|
||||
「把谜题翻译成约束程序并求解」这一核心论点(见下方真实运行结果)。
|
||||
|
||||
### 核心思想：为什么代码辅助更强
|
||||
|
||||
K&K 谜题的关键建模规则只有一条——对每位居民 X 加一条**双条件(等价)约束**：
|
||||
|
||||
```
|
||||
X 是骑士(True) <=> X 说的那句话为真
|
||||
```
|
||||
|
||||
即 `X == (该陈述的语义真值)`。把它交给确定性求解器**穷举**所有布尔组合，逻辑上不会出错；
|
||||
而纯思考在多人、含计数(“恰好两个骑士”)或自指(“我和 B 同类”)的谜题上，很容易在心算
|
||||
真值传播时出错。
|
||||
|
||||
### 文件说明
|
||||
|
||||
| 文件 | 作用 |
|
||||
| --- | --- |
|
||||
| `demo.py` | 主程序：跑 纯思考/代码辅助/约束求解 的对照实验，打印准确率对比表 |
|
||||
| `csp_solver.py` | 离线约束求解器：结构化陈述 DSL + `python-constraint` 求解(供 demo 的 solver 模式与 build_puzzles 校验共用) |
|
||||
| `sandbox.py` | 极简 Code Interpreter：子进程沙箱执行模型生成的 Python(预装 python-constraint) |
|
||||
| `puzzles.json` | 12 道谜题的题面 + 结构化陈述 + 唯一真值解(给 LLM 的只有题面) |
|
||||
| `build_puzzles.py` | 生成/校验谜题：用 `python-constraint` 求解并断言每题“解唯一”，可导出精选题或随机生成 |
|
||||
| `requirements.txt` | 依赖(openai + python-constraint) |
|
||||
| `env.example` | 环境变量样例 |
|
||||
| `last_run.json` | 每次运行后自动保存的逐题完整记录(含模型生成的代码)，便于复盘 |
|
||||
|
||||
### 快速开始
|
||||
|
||||
```bash
|
||||
# 在仓库根目录使用统一的第 5 章环境
|
||||
uv sync --locked --python 3.12 --extra ch5
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch5]"
|
||||
|
||||
cd chapter5/code-for-logic
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
```
|
||||
|
||||
#### 1) 离线约束求解基线（不需要 API Key，推荐先跑）
|
||||
|
||||
```bash
|
||||
python demo.py --mode solver # 用 python-constraint 离线求解全部 12 题
|
||||
python demo.py --mode solver --min-people 4 # 只跑 >=4 人的难题
|
||||
```
|
||||
|
||||
这条路径完全离线、确定性，直接演示「谜题→约束程序→求解」的核心论点，准确率 100%。
|
||||
|
||||
#### 2) LLM 对照实验（需要 OPENAI_API_KEY 或 OPENROUTER_API_KEY）
|
||||
|
||||
```bash
|
||||
cp env.example .env # 然后编辑 .env 填入 OPENAI_API_KEY
|
||||
# 或直接 export OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
python demo.py # 默认 both：纯思考 vs 代码辅助，全部 12 题
|
||||
python demo.py --mode pure # 只跑纯思考基线
|
||||
python demo.py --limit 4 # 只跑前 4 题(省钱冒烟测试)
|
||||
python demo.py --max-people 3 # 只跑 <=3 人的谜题(按难度筛选)
|
||||
python demo.py --model gpt-4o-mini # 指定模型(默认 gpt-4o-mini)
|
||||
python demo.py --puzzles my.json --output run.json # 换数据集/输出路径
|
||||
```
|
||||
|
||||
**通用 OpenRouter 兜底**：未配置 `OPENAI_API_KEY` 时，只要设置了 `OPENROUTER_API_KEY`
|
||||
即自动改走 OpenRouter（`gpt-*` → `openai/*`）。默认模型 `gpt-4o-mini` 是普通 gpt id，可
|
||||
直连 OpenAI；仅当把 `--model` 换成 `gpt-5.x` 这类需组织实名认证的模型、且设置了
|
||||
`OPENROUTER_API_KEY` 时，才会优先走 OpenRouter。
|
||||
|
||||
完整参数见 `python demo.py --help`（中文说明）。
|
||||
|
||||
#### 3) 生成/扩充谜题数据集
|
||||
|
||||
```bash
|
||||
python build_puzzles.py # 导出内置 12 道精选题(默认)
|
||||
python build_puzzles.py --generate 20 --min-people 3 --max-people 5 --seed 7
|
||||
python build_puzzles.py --generate 20 --output my.json
|
||||
```
|
||||
|
||||
随机生成器会用 `python-constraint` 求解每个候选谜题，只保留「解唯一」的题目。
|
||||
|
||||
`sandbox.py` / `csp_solver.py` 也可单独运行做自测：
|
||||
`python sandbox.py`、`python csp_solver.py` 都会用 python-constraint 求解一道最简谜题。
|
||||
|
||||
### 真实运行结果（一）：离线约束求解基线（`--mode solver`，无需 API）
|
||||
|
||||
`python demo.py --mode solver` 的真实输出（12 道精选题，完全离线、确定性）：
|
||||
|
||||
```
|
||||
== 约束求解(solver，离线) ==
|
||||
[solver] kk01 (2人) ✓ 解数=1 预测={'A': 'knight', 'B': 'knave'}
|
||||
[solver] kk05 (3人) ✓ 解数=1 预测={'A': 'knave', 'B': 'knave', 'C': 'knight'}
|
||||
[solver] kk11 (5人) ✓ 解数=1 预测={'A': 'knight', 'B': 'knight', 'C': 'knave', 'D': 'knave', 'E': 'knight'}
|
||||
...（其余题略）
|
||||
------------------------------------------------------------
|
||||
准确率 100.0%
|
||||
============================================================
|
||||
约束求解 准确率: 100.0% (12/12)
|
||||
```
|
||||
|
||||
这条路径把每题的结构化陈述翻译成 `python-constraint` 约束并穷举求解，12/12 全对——
|
||||
它直接证明了「谜题→约束程序→求解」的确定性；LLM 只要把谜题正确翻译成同样的约束，
|
||||
就能拿到同样 100% 的结果（下节）。随机生成的谜题（`build_puzzles.py --generate`）经
|
||||
solver 复核同样 100% 解出且与生成时的唯一解一致。
|
||||
|
||||
### 真实运行结果（二）：LLM 对照实验（gpt-4o-mini，12 题）
|
||||
|
||||
```
|
||||
准确率对比表
|
||||
============================================================
|
||||
题号 人数 纯思考 代码辅助
|
||||
------------------------------------------------------------
|
||||
kk01 2 ✓ ✓
|
||||
kk02 2 ✓ ✓
|
||||
kk03 2 ✓ ✓
|
||||
kk04 3 ✓ ✓
|
||||
kk05 3 ✗ ✓
|
||||
kk06 3 ✗ ✓
|
||||
kk07 3 ✗ ✓
|
||||
kk08 4 ✗ ✓
|
||||
kk09 4 ✗ ✓
|
||||
kk10 4 ✓ ✓
|
||||
kk11 5 ✗ ✓
|
||||
kk12 5 ✓ ✓
|
||||
------------------------------------------------------------
|
||||
准确率 50.0% 100.0%
|
||||
============================================================
|
||||
纯思考 准确率: 50.0% (6/12)
|
||||
代码辅助 准确率: 100.0% (12/12)
|
||||
提升(代码辅助 - 纯思考): +50.0 个百分点
|
||||
```
|
||||
|
||||
> 说明：这里刻意选用能力较弱的 `gpt-4o-mini` 来暴露对照——纯思考只做对了 **6/12
|
||||
> (50%)**，且错误集中在 3 人及以上、含计数/自指的谜题上（kk05~kk09、kk11），正是心算
|
||||
> 真值传播最容易出错的题型；而代码辅助把每句话翻译成双条件约束、交给 `python-constraint`
|
||||
> 穷举求解，**12/12 全对**，一举把准确率拉满，净提升 **+50 个百分点**。这正是本实验想
|
||||
> 说明的核心：把逻辑外包给确定性求解器，正确性不再依赖模型自己的推理强弱。`gpt-4o-mini`
|
||||
> 有一定随机性，多次运行个别题目可能有小幅波动，但“纯思考明显低于代码辅助”的整体格局稳定。
|
||||
|
||||
> **模型与脚手架（harness）是此消彼长的关系**：模型足够强时，脚手架可以更薄——模型自己
|
||||
> 就能算对；模型不够强时，就需要在脚手架里做更多事（如把逻辑交给代码/求解器）来兜住
|
||||
> 正确性。本实验刻意用较弱的 `gpt-4o-mini`，正是为了让这一对照可见——换成 `gpt-5.6-luna`
|
||||
> 这类强推理模型，纯思考也能全解，代码增益会收敛为 0。换句话说，代码辅助（乃至离线
|
||||
> solver）真正的价值，是把正确性变成**确定性、与模型强弱无关**：对更弱的模型或更大/更难
|
||||
> 的谜题，纯思考会随人数增加而掉分，而“翻译成约束程序 + 求解器穷举”的路径始终稳定给出正确解。
|
||||
|
||||
#### 一道谜题的约束建模代码（模型自动生成，kk11，5 人链式+计数）
|
||||
|
||||
题面：A 说“B 是骑士”；B 说“C 是无赖”；C 说“D 是骑士”；D 说“E 是无赖”；
|
||||
E 说“我们五人当中至少有两个骑士”。
|
||||
|
||||
```python
|
||||
from constraint import Problem
|
||||
|
||||
p = Problem()
|
||||
for name in ['A', 'B', 'C', 'D', 'E']:
|
||||
p.addVariable(name, [True, False]) # True=骑士(说真话), False=无赖(说假话)
|
||||
|
||||
# 每句话都写成「X == (那句话的真值)」的双条件约束
|
||||
p.addConstraint(lambda a, b: a == (b == True), ['A', 'B']) # A:"B 是骑士"
|
||||
p.addConstraint(lambda b, c: b == (c == False), ['B', 'C']) # B:"C 是无赖"
|
||||
p.addConstraint(lambda c, d: c == (d == True), ['C', 'D']) # C:"D 是骑士"
|
||||
p.addConstraint(lambda d, e: d == (e == False), ['D', 'E']) # D:"E 是无赖"
|
||||
p.addConstraint(lambda a, b, c, d, e: e == ((a + b + c + d + e) >= 2),
|
||||
['A', 'B', 'C', 'D', 'E']) # E:"至少两个骑士"
|
||||
|
||||
for s in p.getSolutions():
|
||||
print({k: ('knight' if v else 'knave') for k, v in s.items()})
|
||||
# 输出: {'A': 'knight', 'B': 'knight', 'C': 'knave', 'D': 'knave', 'E': 'knight'}
|
||||
```
|
||||
|
||||
求解器直接穷举 2^5=32 种组合，返回满足全部约束的唯一解——这正是纯思考在链式真值
|
||||
传播中最容易算错的题型。
|
||||
|
||||
### 注意事项
|
||||
|
||||
- **成本**：默认 `gpt-4o-mini`（刻意选用较弱模型以显现对照，见上文），跑完 12 题两种模式的开销很小；用 `MODEL`/`--model` 可换更便宜或更强的模型。
|
||||
- **API Key**：从环境变量或 `.env` 读 `OPENAI_API_KEY`（或 `OPENROUTER_API_KEY` 兜底）；用 `MODEL` 可换模型。
|
||||
- **沙箱**：`sandbox.py` 用子进程 + 超时执行代码，属教学用极简沙箱；生产环境应换成
|
||||
容器/gVisor 等更强隔离。
|
||||
- **谜题可靠性**：`build_puzzles.py` 用 `python-constraint` 求解每题(内置精选题或随机生成)，
|
||||
断言“解唯一”后才写出，确保真值解无歧义；想自己加题就改 `CURATED` 或用 `--generate`。
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Run `--mode solver` first for a free offline baseline. / 建议先跑 `--mode solver` 离线基线。
|
||||
- Commands, code, paths, and env vars are identical in both language sections. / 命令、代码、路径与环境变量在中英文两侧保持一致。
|
||||
@@ -0,0 +1,187 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Build the frozen Experiment 5-2 test set from the named Hugging Face dataset.
|
||||
|
||||
The manuscript explicitly names K-and-K/perturbed-knights-and-knaves. This
|
||||
builder downloads a revision-pinned, stratified sample from every test
|
||||
perturbation and every 2--8-person difficulty cell. It retains source identity
|
||||
and hashes and independently checks every published label with the local
|
||||
python-constraint implementation before writing the benchmark JSON.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import hashlib
|
||||
import json
|
||||
import random
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from csp_solver import solve_labeled
|
||||
|
||||
|
||||
DATASET = "K-and-K/perturbed-knights-and-knaves"
|
||||
REVISION = "bc7ee75a15ee8196ccbdb7df3ab46284340412e2"
|
||||
LICENSE = "CC-BY-NC-SA-4.0"
|
||||
PERTURBATIONS = (
|
||||
"perturbed_leaf",
|
||||
"perturbed_statement",
|
||||
"reorder_statement",
|
||||
"random_pair",
|
||||
"uncommon_name",
|
||||
"flip_role",
|
||||
)
|
||||
|
||||
|
||||
def _source_path(perturbation: str, people: int) -> str:
|
||||
return f"test/{perturbation}/people{people}_num100.jsonl"
|
||||
|
||||
|
||||
def _download(path: str) -> bytes:
|
||||
url = (
|
||||
"https://huggingface.co/datasets/"
|
||||
f"{DATASET}/resolve/{REVISION}/{path}?download=true"
|
||||
)
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "ai-agent-book-exp5-2/1.0"})
|
||||
with urllib.request.urlopen(request, timeout=60) as response:
|
||||
return response.read()
|
||||
|
||||
|
||||
def convert_expression(node: Any, names: list[str]) -> list[Any]:
|
||||
"""Convert the dataset's published tuple AST into the lab's JSON DSL."""
|
||||
if not isinstance(node, tuple) or not node:
|
||||
raise ValueError(f"invalid statement AST node: {node!r}")
|
||||
tag = node[0]
|
||||
if tag in {"lying", "telling-truth"}:
|
||||
if len(node) != 2 or not isinstance(node[1], int):
|
||||
raise ValueError(f"invalid identity node: {node!r}")
|
||||
role = "knave" if tag == "lying" else "knight"
|
||||
return ["is", names[node[1]], role]
|
||||
if tag == "not" and len(node) == 2:
|
||||
return ["not", convert_expression(node[1], names)]
|
||||
binary = {"and": "and", "or": "or", "->": "implies", "<=>": "iff"}
|
||||
if tag in binary and len(node) == 3:
|
||||
return [
|
||||
binary[tag],
|
||||
convert_expression(node[1], names),
|
||||
convert_expression(node[2], names),
|
||||
]
|
||||
raise ValueError(f"unsupported statement AST node: {node!r}")
|
||||
|
||||
|
||||
def convert_row(
|
||||
row: dict[str, Any], *, perturbation: str, people: int, source_path: str,
|
||||
source_sha256: str, source_row: int,
|
||||
) -> dict[str, Any]:
|
||||
names = list(row["names"])
|
||||
if len(names) != people:
|
||||
raise ValueError(f"row {source_row}: expected {people} names, got {len(names)}")
|
||||
statements = ast.literal_eval(row["statements"])
|
||||
if not isinstance(statements, tuple) or len(statements) != len(names):
|
||||
raise ValueError(f"row {source_row}: statement count does not match names")
|
||||
structs = {
|
||||
speaker: convert_expression(statement, names)
|
||||
for speaker, statement in zip(names, statements)
|
||||
}
|
||||
gold = {
|
||||
name: ("knight" if truth else "knave")
|
||||
for name, truth in zip(names, row["solution"])
|
||||
}
|
||||
independently_solved = solve_labeled(names, structs)
|
||||
if len(independently_solved) != 1 or independently_solved[0] != gold:
|
||||
raise ValueError(
|
||||
f"row {source_row}: published label failed independent CSP check: "
|
||||
f"gold={gold!r}, solved={independently_solved!r}"
|
||||
)
|
||||
return {
|
||||
"id": f"{perturbation}-p{people}-r{source_row:03d}",
|
||||
"num_people": people,
|
||||
"names": names,
|
||||
"description": row["quiz"],
|
||||
"solution": gold,
|
||||
"statements_struct": structs,
|
||||
"source": {
|
||||
"dataset": DATASET,
|
||||
"revision": REVISION,
|
||||
"license": LICENSE,
|
||||
"config": "test",
|
||||
"split": perturbation,
|
||||
"path": source_path,
|
||||
"file_sha256": source_sha256,
|
||||
"row": source_row,
|
||||
"dataset_index": row.get("index"),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def build(*, per_cell: int, seed: int) -> tuple[list[dict[str, Any]], dict[str, Any]]:
|
||||
if not 1 <= per_cell <= 100:
|
||||
raise ValueError("per_cell must be between 1 and 100")
|
||||
puzzles: list[dict[str, Any]] = []
|
||||
files: list[dict[str, Any]] = []
|
||||
for perturbation in PERTURBATIONS:
|
||||
for people in range(2, 9):
|
||||
path = _source_path(perturbation, people)
|
||||
raw = _download(path)
|
||||
sha256 = hashlib.sha256(raw).hexdigest()
|
||||
rows = [json.loads(line) for line in raw.decode("utf-8").splitlines() if line]
|
||||
if len(rows) < per_cell:
|
||||
raise ValueError(
|
||||
f"{path}: only {len(rows)} published rows, cannot sample {per_cell}"
|
||||
)
|
||||
cell_seed = int.from_bytes(
|
||||
hashlib.sha256(f"{seed}:{path}".encode()).digest()[:8], "big"
|
||||
)
|
||||
indices = sorted(random.Random(cell_seed).sample(range(len(rows)), per_cell))
|
||||
for index in indices:
|
||||
puzzles.append(convert_row(
|
||||
rows[index], perturbation=perturbation, people=people,
|
||||
source_path=path, source_sha256=sha256, source_row=index,
|
||||
))
|
||||
files.append({
|
||||
"path": path,
|
||||
"sha256": sha256,
|
||||
"published_rows": len(rows),
|
||||
"sampled_rows": indices,
|
||||
})
|
||||
manifest = {
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-2",
|
||||
"dataset": DATASET,
|
||||
"revision": REVISION,
|
||||
"license": LICENSE,
|
||||
"sampling": {
|
||||
"split": "test",
|
||||
"perturbations": list(PERTURBATIONS),
|
||||
"people": list(range(2, 9)),
|
||||
"per_cell": per_cell,
|
||||
"seed": seed,
|
||||
"cells": len(PERTURBATIONS) * 7,
|
||||
"total": len(puzzles),
|
||||
},
|
||||
"source_files": files,
|
||||
"label_validation": "all rows independently solved with python-constraint",
|
||||
}
|
||||
return puzzles, manifest
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--per-cell", type=int, default=2)
|
||||
parser.add_argument("--seed", type=int, default=512)
|
||||
parser.add_argument("--output", type=Path, default=Path("hf_test_stratified_84.json"))
|
||||
parser.add_argument("--manifest", type=Path, default=Path("hf_test_stratified_84.manifest.json"))
|
||||
args = parser.parse_args()
|
||||
puzzles, manifest = build(per_cell=args.per_cell, seed=args.seed)
|
||||
args.output.write_text(json.dumps(puzzles, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
args.manifest.write_text(json.dumps(manifest, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
||||
print(json.dumps({
|
||||
"output": str(args.output), "manifest": str(args.manifest),
|
||||
"puzzles": len(puzzles), "revision": REVISION,
|
||||
}, ensure_ascii=False))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,204 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
生成/校验「骑士与无赖」(Knights and Knaves)谜题，并导出 puzzles.json。
|
||||
|
||||
每道谜题的每句话都用 csp_solver.py 里的结构化 DSL 表示(见该文件顶部说明)，
|
||||
既能渲染成中文题面(给 LLM 看)，也能直接翻译成 python-constraint 约束来求解。
|
||||
本脚本用 python-constraint 校验每题「解唯一」后才写出——这确保真值解无歧义，
|
||||
同时演示了实验 5-2 的核心：把谜题形式化为 CSP 并用求解器离线求解。
|
||||
|
||||
约定：骑士(knight)永远说真话，无赖(knave)永远说假话。t[name]=True 表示骑士。
|
||||
|
||||
用法：
|
||||
python build_puzzles.py # 导出内置的 12 道精选谜题(默认)
|
||||
python build_puzzles.py --generate 20 # 随机生成 20 道解唯一的谜题
|
||||
python build_puzzles.py --generate 20 --min-people 3 --max-people 5 --seed 7
|
||||
python build_puzzles.py --output my.json # 指定输出文件
|
||||
"""
|
||||
import argparse
|
||||
import json
|
||||
import random
|
||||
|
||||
from csp_solver import render_nl, solve, solve_labeled
|
||||
|
||||
# 每题：id, 名字列表, dict{name: 结构化陈述}。中文题面由结构化陈述自动渲染，
|
||||
# 但精选题保留手写的更自然的中文(见 STATEMENTS_NL 覆盖)。
|
||||
CURATED = [
|
||||
("kk01", ["A", "B"], {
|
||||
"A": ["is", "B", "knave"],
|
||||
"B": ["and", ["is", "A", "knave"], ["is", "B", "knave"]]}),
|
||||
("kk02", ["A", "B"], {
|
||||
"A": ["same", "A", "B"],
|
||||
"B": ["diff", "A", "B"]}),
|
||||
("kk03", ["A", "B"], {
|
||||
"A": ["count", "knight", ">=", 1],
|
||||
"B": ["is", "A", "knave"]}),
|
||||
("kk04", ["A", "B", "C"], {
|
||||
"A": ["is", "B", "knave"],
|
||||
"B": ["is", "C", "knave"],
|
||||
"C": ["and", ["is", "A", "knave"], ["is", "B", "knave"]]}),
|
||||
("kk05", ["A", "B", "C"], {
|
||||
"A": ["is", "B", "knight"],
|
||||
"B": ["is", "C", "knave"],
|
||||
"C": ["same", "A", "B"]}),
|
||||
("kk06", ["A", "B", "C"], {
|
||||
"A": ["same", "B", "C"],
|
||||
"B": ["is", "A", "knave"],
|
||||
"C": ["same", "C", "A"]}),
|
||||
("kk07", ["A", "B", "C"], {
|
||||
"A": ["or", ["is", "A", "knave"], ["is", "B", "knight"]],
|
||||
"B": ["is", "A", "knight"],
|
||||
"C": ["is", "B", "knave"]}),
|
||||
("kk08", ["A", "B", "C", "D"], {
|
||||
"A": ["same", "B", "D"],
|
||||
"B": ["is", "C", "knave"],
|
||||
"C": ["is", "D", "knight"],
|
||||
"D": ["diff", "B", "C"]}),
|
||||
("kk09", ["A", "B", "C", "D"], {
|
||||
"A": ["is", "B", "knight"],
|
||||
"B": ["is", "C", "knave"],
|
||||
"C": ["is", "D", "knight"],
|
||||
"D": ["diff", "A", "B"]}),
|
||||
("kk10", ["A", "B", "C", "D"], {
|
||||
"A": ["count", "knave", ">=", 3],
|
||||
"B": ["is", "A", "knave"],
|
||||
"C": ["is", "B", "knight"],
|
||||
"D": ["is", "C", "knave"]}),
|
||||
("kk11", ["A", "B", "C", "D", "E"], {
|
||||
"A": ["is", "B", "knight"],
|
||||
"B": ["is", "C", "knave"],
|
||||
"C": ["is", "D", "knight"],
|
||||
"D": ["is", "E", "knave"],
|
||||
"E": ["count", "knight", ">=", 2]}),
|
||||
("kk12", ["A", "B", "C", "D", "E"], {
|
||||
"A": ["is", "B", "knight"],
|
||||
"B": ["is", "C", "knave"],
|
||||
"C": ["is", "D", "knave"],
|
||||
"D": ["is", "E", "knight"],
|
||||
"E": ["same", "A", "C"]}),
|
||||
]
|
||||
|
||||
# 精选题的手写中文题面(比自动渲染更自然)。未覆盖的句子回退到 render_nl。
|
||||
STATEMENTS_NL = {
|
||||
("kk01", "B"): "我们两人都不是骑士。",
|
||||
("kk02", "A"): "我和 B 是同一类人（要么都是骑士，要么都是无赖）。",
|
||||
("kk02", "B"): "我和 A 是不同类人。",
|
||||
("kk03", "A"): "我们当中至少有一个骑士。",
|
||||
("kk04", "C"): "A 和 B 都是无赖。",
|
||||
("kk06", "C"): "我和 A 是同一类人。",
|
||||
("kk07", "A"): "我是无赖，或者 B 是骑士。",
|
||||
("kk09", "D"): "A 和 B 不是同一类人。",
|
||||
("kk10", "A"): "我们四人当中至少有三个无赖。",
|
||||
("kk11", "E"): "我们五人当中至少有两个骑士。",
|
||||
("kk12", "E"): "A 和 C 是同一类人。",
|
||||
}
|
||||
|
||||
|
||||
def build_puzzle(pid, names, structs, nl_overrides=None):
|
||||
"""求解校验(要求解唯一)并组装成写入 puzzles.json 的一条记录。"""
|
||||
sols = solve_labeled(names, structs)
|
||||
if len(sols) != 1:
|
||||
raise ValueError(f"{pid} 解不唯一: {len(sols)} 个解 -> {sols}")
|
||||
solution = sols[0]
|
||||
|
||||
nl_overrides = nl_overrides or {}
|
||||
statements = {n: nl_overrides.get(n, render_nl(structs[n])) for n in names}
|
||||
lines = [f"{n}: 「{statements[n]}」" for n in names]
|
||||
desc = (
|
||||
f"这座岛上有 {len(names)} 位居民：{', '.join(names)}。"
|
||||
"每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。"
|
||||
"他们各自说了如下的话：\n" + "\n".join(lines)
|
||||
)
|
||||
return dict(id=pid, num_people=len(names), names=names,
|
||||
statements=statements, statements_struct=structs,
|
||||
description=desc, solution=solution)
|
||||
|
||||
|
||||
# ---------------- 随机生成器 ----------------
|
||||
def _random_stmt(speaker, names, rng):
|
||||
"""为 speaker 随机生成一句合法的结构化陈述。"""
|
||||
others = [n for n in names if n != speaker]
|
||||
# Solo resident: only count statements are valid (no "others" to name).
|
||||
if not others:
|
||||
role = rng.choice(["knight", "knave"])
|
||||
op = rng.choice([">=", "<=", "=="])
|
||||
return ["count", role, op, rng.randint(1, len(names))]
|
||||
kind = rng.choice(["is", "is", "same", "diff", "count"])
|
||||
if kind == "is":
|
||||
return ["is", rng.choice(others), rng.choice(["knight", "knave"])]
|
||||
if kind == "same":
|
||||
return ["same", speaker, rng.choice(others)]
|
||||
if kind == "diff":
|
||||
return ["diff", speaker, rng.choice(others)]
|
||||
# count：全体中某角色的人数满足某比较
|
||||
role = rng.choice(["knight", "knave"])
|
||||
op = rng.choice([">=", "<=", "=="])
|
||||
k = rng.randint(1, len(names))
|
||||
return ["count", role, op, k]
|
||||
|
||||
|
||||
def generate(count, min_people, max_people, seed):
|
||||
"""随机生成 count 道「解唯一」的谜题(用 python-constraint 过滤)。"""
|
||||
rng = random.Random(seed)
|
||||
names_pool = ["A", "B", "C", "D", "E", "F", "G"]
|
||||
puzzles = []
|
||||
attempts = 0
|
||||
while len(puzzles) < count and attempts < count * 2000:
|
||||
attempts += 1
|
||||
n = rng.randint(min_people, max_people)
|
||||
names = names_pool[:n]
|
||||
structs = {sp: _random_stmt(sp, names, rng) for sp in names}
|
||||
if len(solve(names, structs)) != 1: # 只保留解唯一的谜题
|
||||
continue
|
||||
pid = f"gen{len(puzzles) + 1:03d}"
|
||||
puzzles.append(build_puzzle(pid, names, structs))
|
||||
if len(puzzles) < count:
|
||||
print(f"警告：{attempts} 次尝试只生成了 {len(puzzles)}/{count} 道解唯一的谜题。")
|
||||
return puzzles
|
||||
|
||||
|
||||
def build_curated():
|
||||
out = []
|
||||
for pid, names, structs in CURATED:
|
||||
nl = {n: STATEMENTS_NL[(pid, n)]
|
||||
for n in names if (pid, n) in STATEMENTS_NL}
|
||||
out.append(build_puzzle(pid, names, structs, nl))
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(
|
||||
description="生成/校验骑士与无赖谜题并导出 puzzles.json"
|
||||
"（用 python-constraint 离线求解，校验每题解唯一）",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=__doc__)
|
||||
ap.add_argument("--generate", type=int, metavar="N", default=0,
|
||||
help="随机生成 N 道解唯一的谜题(默认 0=导出内置 12 道精选题)")
|
||||
ap.add_argument("--min-people", type=int, default=2,
|
||||
help="随机生成时每题最少居民数(默认 2)")
|
||||
ap.add_argument("--max-people", type=int, default=5,
|
||||
help="随机生成时每题最多居民数(难度上限，默认 5)")
|
||||
ap.add_argument("--seed", type=int, default=42,
|
||||
help="随机种子，保证可复现(默认 42)")
|
||||
ap.add_argument("--output", default="puzzles.json",
|
||||
help="输出文件路径(默认 puzzles.json)")
|
||||
args = ap.parse_args()
|
||||
|
||||
if args.generate > 0:
|
||||
print(f"随机生成 {args.generate} 道谜题"
|
||||
f"({args.min_people}~{args.max_people} 人，seed={args.seed})...")
|
||||
out = generate(args.generate, args.min_people, args.max_people, args.seed)
|
||||
else:
|
||||
out = build_curated()
|
||||
|
||||
for p in out:
|
||||
print(f"{p['id']}: OK 唯一解 = {p['solution']}")
|
||||
|
||||
with open(args.output, "w", encoding="utf-8") as f:
|
||||
json.dump(out, f, ensure_ascii=False, indent=2)
|
||||
print(f"\n已写出 {len(out)} 题到 {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,127 @@
|
||||
"""
|
||||
离线约束求解器：把「骑士与无赖」谜题的结构化陈述翻译成约束满足问题(CSP)，
|
||||
用 python-constraint 库求解——这是实验 5-2 想论证的「代码求解」路径的确定性参考实现。
|
||||
|
||||
它不依赖任何 LLM / 网络，可完全离线运行，因此既用于 build_puzzles.py 校验谜题
|
||||
「解唯一」，也用于 demo.py 的 solver 模式给出约束求解基线(理论上 100% 正确)。
|
||||
|
||||
【结构化陈述 DSL】每句话用一个 JSON 可序列化的列表表示，节点形式如下
|
||||
(True=骑士/说真话，False=无赖/说假话)：
|
||||
|
||||
["is", target, "knight"|"knave"] # target 是骑士 / 无赖
|
||||
["same", a, b] # a 和 b 是同一类人
|
||||
["diff", a, b] # a 和 b 是不同类人
|
||||
["count", "knight"|"knave", op, k] # 全体中该角色的人数 op k, op ∈ {">=","<=","=="}
|
||||
["and", s1, s2] # 合取
|
||||
["or", s1, s2] # 析取
|
||||
["not", s1] # 否定
|
||||
["implies", s1, s2] # 蕴含 s1 -> s2
|
||||
["iff", s1, s2] # 双条件 s1 <-> s2
|
||||
|
||||
关键建模规则：对每位说话者 X 加一条【双条件约束】 `t[X] == eval_stmt(X 的话)`——
|
||||
X 是骑士当且仅当他的话为真。绝不能把话本身当作硬约束。
|
||||
"""
|
||||
from constraint import Problem
|
||||
|
||||
_OPS = {">=": lambda a, b: a >= b,
|
||||
"<=": lambda a, b: a <= b,
|
||||
"==": lambda a, b: a == b}
|
||||
|
||||
|
||||
def eval_stmt(node, t):
|
||||
"""在赋值 t(name->bool, True=骑士) 下求某句话的语义真值。"""
|
||||
tag = node[0]
|
||||
if tag == "is":
|
||||
_, target, role = node
|
||||
return t[target] if role == "knight" else (not t[target])
|
||||
if tag == "same":
|
||||
return t[node[1]] == t[node[2]]
|
||||
if tag == "diff":
|
||||
return t[node[1]] != t[node[2]]
|
||||
if tag == "count":
|
||||
_, role, op, k = node
|
||||
want = (role == "knight")
|
||||
cnt = sum(1 for v in t.values() if v == want)
|
||||
return _OPS[op](cnt, k)
|
||||
if tag == "and":
|
||||
return eval_stmt(node[1], t) and eval_stmt(node[2], t)
|
||||
if tag == "or":
|
||||
return eval_stmt(node[1], t) or eval_stmt(node[2], t)
|
||||
if tag == "not":
|
||||
return not eval_stmt(node[1], t)
|
||||
if tag == "implies":
|
||||
return (not eval_stmt(node[1], t)) or eval_stmt(node[2], t)
|
||||
if tag == "iff":
|
||||
return eval_stmt(node[1], t) == eval_stmt(node[2], t)
|
||||
raise ValueError(f"未知的陈述节点: {node!r}")
|
||||
|
||||
|
||||
def solve(names, structs):
|
||||
"""用 python-constraint 求解，返回所有满足约束的赋值(dict name->bool)列表。
|
||||
|
||||
names : 居民名字列表
|
||||
structs : dict name -> 该居民陈述的结构化 DSL
|
||||
"""
|
||||
problem = Problem()
|
||||
for n in names:
|
||||
problem.addVariable(n, [True, False])
|
||||
|
||||
# 对每位说话者加一条双条件约束：t[X] == (X 的话为真)
|
||||
for speaker in names:
|
||||
stmt = structs.get(speaker)
|
||||
if stmt is None:
|
||||
continue
|
||||
def make_constraint(speaker=speaker, stmt=stmt):
|
||||
def constraint(*values):
|
||||
t = dict(zip(names, values))
|
||||
return t[speaker] == eval_stmt(stmt, t)
|
||||
return constraint
|
||||
|
||||
problem.addConstraint(make_constraint(), names)
|
||||
|
||||
return problem.getSolutions()
|
||||
|
||||
|
||||
def solve_labeled(names, structs):
|
||||
"""求解并把布尔解转成 {name: 'knight'/'knave'}。返回解列表(通常唯一)。"""
|
||||
out = []
|
||||
for sol in solve(names, structs):
|
||||
out.append({n: ("knight" if sol[n] else "knave") for n in names})
|
||||
return out
|
||||
|
||||
|
||||
def render_nl(node):
|
||||
"""把结构化陈述渲染成中文题面(供随机生成的谜题使用)。"""
|
||||
tag = node[0]
|
||||
if tag == "is":
|
||||
role = "骑士" if node[2] == "knight" else "无赖"
|
||||
return f"{node[1]} 是{role}。"
|
||||
if tag == "same":
|
||||
return f"{node[1]} 和 {node[2]} 是同一类人。"
|
||||
if tag == "diff":
|
||||
return f"{node[1]} 和 {node[2]} 是不同类人。"
|
||||
if tag == "count":
|
||||
role = "骑士" if node[1] == "knight" else "无赖"
|
||||
word = {">=": "至少", "<=": "至多", "==": "恰好"}[node[2]]
|
||||
return f"我们当中{word}有 {node[3]} 个{role}。"
|
||||
if tag == "and":
|
||||
return f"{render_nl(node[1])[:-1]}，并且 {render_nl(node[2])}"
|
||||
if tag == "or":
|
||||
return f"{render_nl(node[1])[:-1]}，或者 {render_nl(node[2])}"
|
||||
if tag == "not":
|
||||
return f"以下说法不成立：{render_nl(node[1])}"
|
||||
if tag == "implies":
|
||||
return f"如果 {render_nl(node[1])[:-1]}，那么 {render_nl(node[2])}"
|
||||
if tag == "iff":
|
||||
return f"{render_nl(node[1])[:-1]}，当且仅当 {render_nl(node[2])}"
|
||||
raise ValueError(f"未知的陈述节点: {node!r}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 自测：kk01 —— A 说"B 是无赖"，B 说"我们都不是骑士"
|
||||
names = ["A", "B"]
|
||||
structs = {
|
||||
"A": ["is", "B", "knave"],
|
||||
"B": ["and", ["is", "A", "knave"], ["is", "B", "knave"]],
|
||||
}
|
||||
print("求解结果:", solve_labeled(names, structs))
|
||||
@@ -0,0 +1,640 @@
|
||||
#!/usr/bin/env python3
|
||||
# -*- coding: utf-8 -*-
|
||||
"""
|
||||
实验 5-2：用代码生成工具提升逻辑思考能力
|
||||
|
||||
对比在三种模式下求解「骑士与无赖」(Knights & Knaves) 谜题的准确率：
|
||||
|
||||
1) 纯思考(pure) —— LLM 仅靠自然语言链式推理直接给出答案；
|
||||
2) 代码辅助(code) —— LLM 配备 Code Interpreter(预装 python-constraint)，
|
||||
把谜题形式化为约束满足问题(CSP)，调用求解器搜索答案；
|
||||
3) 约束求解(solver) —— 【离线，无需 API】直接用 python-constraint 求解结构化
|
||||
陈述，作为确定性基线(理论上 100% 正确)。
|
||||
|
||||
结论预期：约束求解把逻辑推理外包给确定性求解器，准确率应达 90%+，
|
||||
且显著高于纯思考模式(纯思考在多人、含计数/自指的谜题上容易出错)。
|
||||
|
||||
用法：
|
||||
# 离线约束求解基线(不花钱、不联网，演示核心论点)：
|
||||
python demo.py --mode solver
|
||||
|
||||
# LLM 对照实验(需要 OPENAI_API_KEY)：
|
||||
export OPENAI_API_KEY=your-openai-api-key
|
||||
python demo.py # 默认 both：跑 纯思考 vs 代码辅助 全部题目
|
||||
python demo.py --mode pure # 只跑纯思考
|
||||
python demo.py --limit 4 # 只跑前 4 题(省钱冒烟测试)
|
||||
python demo.py --max-people 3 # 只跑不超过 3 人的谜题(按难度筛选)
|
||||
python demo.py --model gpt-4o-mini # 指定模型(默认 gpt-4o-mini)
|
||||
python demo.py --puzzles my.json # 换一份谜题数据集
|
||||
"""
|
||||
import argparse
|
||||
import datetime as dt
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from csp_solver import solve_labeled
|
||||
from sandbox import run_python
|
||||
|
||||
# ---- 读取 .env(如果存在)。避免额外依赖，手写一个极简解析器。----
|
||||
def _load_dotenv(path=".env"):
|
||||
if not os.path.exists(path):
|
||||
return
|
||||
with open(path, encoding="utf-8") as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line or line.startswith("#") or "=" not in line:
|
||||
continue
|
||||
k, v = line.split("=", 1)
|
||||
os.environ.setdefault(k.strip(), v.strip().strip('"').strip("'"))
|
||||
|
||||
|
||||
_load_dotenv()
|
||||
|
||||
MODEL = os.environ.get("MODEL", "gpt-4o-mini")
|
||||
PROVIDER = "unknown"
|
||||
|
||||
# --- 通用 OpenRouter 兜底：无直连 key 时自动改走 OpenRouter ---
|
||||
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
|
||||
|
||||
def map_model_to_openrouter(model: str) -> str:
|
||||
"""把直连模型名映射为 OpenRouter 上的 id（非可映射 id 统一兜底到当前廉价旗舰）。"""
|
||||
if not model or "/" in model:
|
||||
return model or "openai/gpt-5.6-luna"
|
||||
m = model.lower()
|
||||
if m.startswith(("gpt-", "o1", "o3", "o4")):
|
||||
return "openai/" + model
|
||||
if m.startswith("claude"):
|
||||
if "haiku" in m:
|
||||
return "anthropic/claude-haiku-4.5"
|
||||
if "sonnet" in m:
|
||||
return "anthropic/claude-sonnet-4.6"
|
||||
return "anthropic/claude-opus-4.8"
|
||||
if m.startswith("gemini"):
|
||||
return "google/" + model
|
||||
return "openai/gpt-5.6-luna"
|
||||
|
||||
|
||||
def build_client_and_model(provider="auto"):
|
||||
"""构造 OpenAI 客户端并返回 (client, model)。
|
||||
|
||||
- 有 OPENAI_API_KEY：直连（默认模型 gpt-4o-mini 是普通 gpt id，可直连 OpenAI）。
|
||||
仅当模型是 gpt-5.x 且同时设置了 OPENROUTER_API_KEY 时才优先走 OpenRouter
|
||||
（直连 gpt-5.x 需组织实名认证）。
|
||||
- 无 OPENAI_API_KEY 但有 OPENROUTER_API_KEY：整体改走 OpenRouter。
|
||||
"""
|
||||
from openai import OpenAI
|
||||
global MODEL, PROVIDER
|
||||
choices = {
|
||||
"ollama": ("ollama", os.environ.get(
|
||||
"OLLAMA_BASE_URL", "http://127.0.0.1:11434/v1"
|
||||
), MODEL),
|
||||
"openai": (os.environ.get("OPENAI_API_KEY"),
|
||||
os.environ.get("OPENAI_BASE_URL"), MODEL),
|
||||
"openrouter": (os.environ.get("OPENROUTER_API_KEY"),
|
||||
OPENROUTER_BASE_URL, map_model_to_openrouter(MODEL)),
|
||||
"moonshot": (os.environ.get("MOONSHOT_API_KEY"),
|
||||
"https://api.moonshot.cn/v1", MODEL),
|
||||
"ark": (os.environ.get("ARK_API_KEY"),
|
||||
"https://ark.cn-beijing.volces.com/api/v3", MODEL),
|
||||
}
|
||||
if provider == "auto":
|
||||
provider = next(
|
||||
(name for name in ("openai", "openrouter", "moonshot", "ark")
|
||||
if choices[name][0]),
|
||||
"openai",
|
||||
)
|
||||
if provider not in choices:
|
||||
raise ValueError(f"unsupported provider: {provider}")
|
||||
api_key, base_url, MODEL = choices[provider]
|
||||
if not api_key:
|
||||
raise SystemExit(f"错误：provider={provider} 缺少对应 API key")
|
||||
PROVIDER = provider
|
||||
kw = {"api_key": api_key, "timeout": 180.0, "max_retries": 5}
|
||||
if base_url:
|
||||
kw["base_url"] = base_url
|
||||
return OpenAI(**kw), MODEL
|
||||
|
||||
|
||||
def _reasoning(model: str) -> bool:
|
||||
"""推理模型（gpt-5 / o 系列 / *thinking 等）不接受 temperature=0。"""
|
||||
return any(k in (model or "").lower()
|
||||
for k in ("gpt-5", "o1", "o3", "o4", "thinking", "reasoner", "kimi-k3"))
|
||||
|
||||
# run_python 工具的 function calling 定义
|
||||
TOOLS = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "run_python",
|
||||
"description": (
|
||||
"在预装了 python-constraint 库的沙箱中执行 Python 代码，返回 stdout/stderr。"
|
||||
"用它把逻辑谜题建模为约束满足问题并求解。记得用 print() 打印结果。"
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"code": {"type": "string", "description": "要执行的完整 Python 代码"}
|
||||
},
|
||||
"required": ["code"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
|
||||
ANSWER_HINT = (
|
||||
'推理结束后，请在最后单独用一行输出 JSON 形式的最终答案，'
|
||||
'键为每个居民的名字，值为 "knight" 或 "knave"，例如：'
|
||||
'{"A": "knight", "B": "knave"}'
|
||||
)
|
||||
|
||||
PURE_SYSTEM = (
|
||||
"你是逻辑推理专家。在「骑士与无赖」谜题中，骑士永远说真话，无赖永远说假话。"
|
||||
"请仅凭自己的推理，逐步分析每位居民的身份，找出满足所有陈述的唯一解。\n" + ANSWER_HINT
|
||||
)
|
||||
|
||||
CODE_SYSTEM = (
|
||||
"你是逻辑推理专家，擅长把谜题转化为形式化约束并用代码求解。"
|
||||
"在「骑士与无赖」谜题中，骑士永远说真话，无赖永远说假话。\n"
|
||||
"请务必使用 run_python 工具，用 python-constraint 库把谜题建模为约束满足问题(CSP)来求解。\n\n"
|
||||
"【最关键的建模规则】不要把某人的陈述直接当成事实约束！"
|
||||
"正确做法是对每位居民 X 加一条【双条件(等价)约束】：\n"
|
||||
" X 的布尔值 == (X 那句话在语义上为真)\n"
|
||||
"含义：X 是骑士(True) 当且仅当 他的话为真；X 是无赖(False) 当且仅当 他的话为假。\n"
|
||||
"这条规则对每一句话都适用，包括计数类('恰好有两个骑士')和自指类('我和 B 同类')的陈述——"
|
||||
"都要写成 `X == (那句话的真值表达式)`，绝不能把 `(那句话的真值表达式)` 单独当作硬约束。\n\n"
|
||||
"示例(设 True=骑士)：\n"
|
||||
" from constraint import Problem\n"
|
||||
" p = Problem()\n"
|
||||
" for name in ['A','B','C']:\n"
|
||||
" p.addVariable(name, [True, False])\n"
|
||||
" # A 说'我们中恰好有一个骑士' -> A == ( (A+B+C)==1 )\n"
|
||||
" p.addConstraint(lambda a,b,c: a == ((a+b+c)==1), ['A','B','C'])\n"
|
||||
" # B 说'C 是无赖' -> B == (not C)\n"
|
||||
" p.addConstraint(lambda b,c: b == (not c), ['B','C'])\n"
|
||||
" # C 说'我和 A 是同一类人' -> C == (C == A)\n"
|
||||
" p.addConstraint(lambda a,c: c == (c == a), ['A','C'])\n"
|
||||
" for s in p.getSolutions():\n"
|
||||
" print({k:('knight' if v else 'knave') for k,v in s.items()})\n\n"
|
||||
"步骤：1) 每人一个布尔变量；2) 每句话写成上面的双条件约束；"
|
||||
"3) 调用 getSolutions() 枚举所有解并 print。\n"
|
||||
"最终答案必须严格采用求解器打印出的解，不要用自己的直觉去推翻它。"
|
||||
"若求解器输出为空，说明约束建错了(很可能漏了双条件)，请检查并重跑。\n" + ANSWER_HINT
|
||||
)
|
||||
|
||||
|
||||
def parse_answer(text, names):
|
||||
"""从模型输出里提取最后一个形如 {name: knight/knave} 的 JSON 答案。"""
|
||||
norm = {
|
||||
"knight": "knight",
|
||||
"knave": "knave",
|
||||
"骑士": "knight",
|
||||
"无赖": "knave",
|
||||
"true": "knight",
|
||||
"false": "knave",
|
||||
"1": "knight",
|
||||
"0": "knave",
|
||||
}
|
||||
# 找出所有 {...} 片段，从后往前尝试解析
|
||||
for m in reversed(list(re.finditer(r"\{[^{}]*\}", text))):
|
||||
try:
|
||||
obj = json.loads(m.group(0))
|
||||
except json.JSONDecodeError:
|
||||
try:
|
||||
import ast
|
||||
obj = ast.literal_eval(m.group(0))
|
||||
except (SyntaxError, ValueError):
|
||||
continue
|
||||
if not isinstance(obj, dict):
|
||||
continue
|
||||
got = {}
|
||||
for n in names:
|
||||
if n not in obj:
|
||||
break
|
||||
v = str(obj[n]).strip().lower()
|
||||
v = norm.get(v, norm.get(str(obj[n]).strip(), None))
|
||||
if v is None:
|
||||
break
|
||||
got[n] = v
|
||||
else:
|
||||
return got
|
||||
return None
|
||||
|
||||
|
||||
def call_model(client, system, user, use_tools):
|
||||
"""Run one trajectory and retain credential-free provider receipts."""
|
||||
messages = [{"role": "system", "content": system},
|
||||
{"role": "user", "content": user}]
|
||||
codes = []
|
||||
receipts = []
|
||||
for turn in range(8): # 最多 8 轮，防止无限循环
|
||||
kwargs = (dict(model=MODEL, messages=messages, temperature=1, max_tokens=8192)
|
||||
if _reasoning(MODEL)
|
||||
else dict(model=MODEL, messages=messages, temperature=0))
|
||||
if use_tools:
|
||||
kwargs.update(
|
||||
tools=TOOLS,
|
||||
tool_choice="required" if not codes else "auto",
|
||||
)
|
||||
resp = client.chat.completions.create(**kwargs)
|
||||
msg = resp.choices[0].message
|
||||
usage = getattr(resp, "usage", None)
|
||||
receipts.append({
|
||||
"turn": turn + 1,
|
||||
"response_id": getattr(resp, "id", None),
|
||||
"response_model": getattr(resp, "model", None),
|
||||
"finish_reason": getattr(resp.choices[0], "finish_reason", None),
|
||||
"usage": {
|
||||
"prompt_tokens": getattr(usage, "prompt_tokens", None),
|
||||
"completion_tokens": getattr(usage, "completion_tokens", None),
|
||||
"total_tokens": getattr(usage, "total_tokens", None),
|
||||
"cached_prompt_tokens": getattr(
|
||||
getattr(usage, "prompt_tokens_details", None),
|
||||
"cached_tokens", None,
|
||||
),
|
||||
},
|
||||
"tool_calls": len(getattr(msg, "tool_calls", None) or []),
|
||||
})
|
||||
if use_tools and msg.tool_calls:
|
||||
messages.append(msg)
|
||||
for tc in msg.tool_calls:
|
||||
try:
|
||||
code = json.loads(tc.function.arguments).get("code", "")
|
||||
except json.JSONDecodeError:
|
||||
code = ""
|
||||
codes.append(code)
|
||||
result = run_python(code)
|
||||
messages.append({"role": "tool", "tool_call_id": tc.id,
|
||||
"content": result})
|
||||
continue
|
||||
return msg.content or "", codes, receipts
|
||||
return "", codes, receipts
|
||||
|
||||
|
||||
def run_mode(client, puzzles, mode, existing=None, checkpoint=None):
|
||||
"""跑一种 LLM 模式(pure/code)，返回逐题记录列表。"""
|
||||
system = CODE_SYSTEM if mode == "code" else PURE_SYSTEM
|
||||
existing_by_id = {
|
||||
record["id"]: record for record in (existing or [])
|
||||
if record.get("id")
|
||||
}
|
||||
records = []
|
||||
for p in puzzles:
|
||||
previous = existing_by_id.get(p["id"])
|
||||
if (
|
||||
previous
|
||||
and not previous.get("provider_error")
|
||||
and previous.get("provider_receipts")
|
||||
and (mode != "code" or previous.get("codes"))
|
||||
):
|
||||
record = previous
|
||||
else:
|
||||
started = time.monotonic()
|
||||
try:
|
||||
text, codes, receipts = call_model(
|
||||
client, system, p["description"], mode == "code"
|
||||
)
|
||||
pred = parse_answer(text, p["names"])
|
||||
record = dict(
|
||||
id=p["id"], num=p["num_people"], pred=pred,
|
||||
gold=p["solution"], correct=pred == p["solution"],
|
||||
source=p.get("source"), codes=codes, text=text,
|
||||
used_python_constraint=any(
|
||||
re.search(r"(^|\s)(from|import)\s+constraint\b", code)
|
||||
for code in codes
|
||||
),
|
||||
duration_s=round(time.monotonic() - started, 3),
|
||||
provider_receipts=receipts,
|
||||
provider_error=None,
|
||||
)
|
||||
except Exception as exc:
|
||||
record = dict(
|
||||
id=p["id"], num=p["num_people"], pred=None,
|
||||
gold=p["solution"], correct=False,
|
||||
source=p.get("source"), codes=[], text="",
|
||||
used_python_constraint=False,
|
||||
duration_s=round(time.monotonic() - started, 3),
|
||||
provider_receipts=[],
|
||||
provider_error=f"{type(exc).__name__}: {exc}",
|
||||
)
|
||||
records.append(record)
|
||||
if checkpoint:
|
||||
checkpoint(mode, records)
|
||||
pred = record.get("pred")
|
||||
correct = bool(record.get("correct"))
|
||||
mark = "✓" if correct else "✗"
|
||||
print(f" [{mode:6}] {p['id']} ({p['num_people']}人) {mark} "
|
||||
f"预测={pred}")
|
||||
return records
|
||||
|
||||
|
||||
def run_solver(puzzles):
|
||||
"""离线约束求解模式：直接用 python-constraint 求解结构化陈述，无需 LLM/API。"""
|
||||
records = []
|
||||
for p in puzzles:
|
||||
struct = p.get("statements_struct")
|
||||
if not struct:
|
||||
sys.exit(f"错误：谜题 {p['id']} 缺少 statements_struct 字段，"
|
||||
"请用新版 build_puzzles.py 重新生成 puzzles.json。")
|
||||
sols = solve_labeled(p["names"], struct)
|
||||
pred = sols[0] if len(sols) == 1 else None
|
||||
correct = pred == p["solution"]
|
||||
records.append(dict(id=p["id"], num=p["num_people"], pred=pred,
|
||||
gold=p["solution"], correct=correct,
|
||||
codes=[], text="", num_solutions=len(sols)))
|
||||
mark = "✓" if correct else "✗"
|
||||
print(f" [solver] {p['id']} ({p['num_people']}人) {mark} "
|
||||
f"解数={len(sols)} 预测={pred}")
|
||||
return records
|
||||
|
||||
|
||||
LABELS = {"pure": "纯思考", "code": "代码辅助", "solver": "约束求解"}
|
||||
|
||||
|
||||
def _wilson(successes, total, z=1.959963984540054):
|
||||
if total <= 0:
|
||||
return [None, None]
|
||||
p = successes / total
|
||||
denominator = 1 + z * z / total
|
||||
center = (p + z * z / (2 * total)) / denominator
|
||||
half = z * math.sqrt(p * (1 - p) / total + z * z / (4 * total * total)) / denominator
|
||||
return [center - half, center + half]
|
||||
|
||||
|
||||
def paired_statistics(pure, code):
|
||||
"""Preregistered paired accuracy analysis (exact McNemar/binomial test)."""
|
||||
if [r["id"] for r in pure] != [r["id"] for r in code]:
|
||||
raise ValueError("paired modes do not contain the same ordered task ids")
|
||||
pure_only = sum(a["correct"] and not b["correct"] for a, b in zip(pure, code))
|
||||
code_only = sum(not a["correct"] and b["correct"] for a, b in zip(pure, code))
|
||||
discordant = pure_only + code_only
|
||||
if discordant:
|
||||
tail = sum(math.comb(discordant, i) for i in range(min(pure_only, code_only) + 1))
|
||||
p_value = min(1.0, 2 * tail / (2 ** discordant))
|
||||
else:
|
||||
p_value = 1.0
|
||||
code_ok = sum(r["correct"] for r in code)
|
||||
pure_ok = sum(r["correct"] for r in pure)
|
||||
code_accuracy = code_ok / len(code)
|
||||
pure_accuracy = pure_ok / len(pure)
|
||||
library_rate = sum(r["used_python_constraint"] for r in code) / len(code)
|
||||
return {
|
||||
"test": "two-sided exact McNemar/binomial test on discordant pairs",
|
||||
"n": len(code),
|
||||
"contingency": {"pure_only": pure_only, "code_only": code_only,
|
||||
"discordant": discordant},
|
||||
"pure_accuracy": pure_accuracy,
|
||||
"code_accuracy": code_accuracy,
|
||||
"accuracy_delta": code_accuracy - pure_accuracy,
|
||||
"code_accuracy_wilson_95": _wilson(code_ok, len(code)),
|
||||
"p_value": p_value,
|
||||
"python_constraint_tool_use_rate": library_rate,
|
||||
"acceptance": {
|
||||
"code_accuracy_over_90_percent": code_accuracy > 0.90,
|
||||
"code_significantly_higher_than_pure": (
|
||||
code_accuracy > pure_accuracy and p_value < 0.05
|
||||
),
|
||||
"all_code_trajectories_used_python_constraint": library_rate == 1.0,
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def campaign_completion(results, puzzles, manifest, mode):
|
||||
"""Check exact protocol execution without requiring a positive result."""
|
||||
expected_ids = [puzzle["id"] for puzzle in puzzles]
|
||||
manifest_exact = bool(
|
||||
manifest
|
||||
and manifest.get("dataset") == "K-and-K/perturbed-knights-and-knaves"
|
||||
and manifest.get("revision")
|
||||
== "bc7ee75a15ee8196ccbdb7df3ab46284340412e2"
|
||||
and (manifest.get("sampling") or {}).get("total") == 84
|
||||
and (manifest.get("sampling") or {}).get("cells") == 42
|
||||
and (manifest.get("sampling") or {}).get("per_cell") == 2
|
||||
and len(manifest.get("source_files") or []) == 42
|
||||
and manifest.get("label_validation")
|
||||
== "all rows independently solved with python-constraint"
|
||||
)
|
||||
exact_coverage = (
|
||||
len(expected_ids) == 84 and len(set(expected_ids)) == 84
|
||||
)
|
||||
errors = []
|
||||
arm_checks = {}
|
||||
for arm in ("pure", "code"):
|
||||
records = results.get(arm) or []
|
||||
arm_errors = [
|
||||
{"id": record.get("id"), "error": record.get("provider_error")}
|
||||
for record in records if record.get("provider_error")
|
||||
]
|
||||
errors.extend({"arm": arm, **error} for error in arm_errors)
|
||||
arm_checks[f"all_{arm}_trajectories_complete"] = (
|
||||
len(records) == 84
|
||||
and [record.get("id") for record in records] == expected_ids
|
||||
and not arm_errors
|
||||
and all(record.get("provider_receipts") for record in records)
|
||||
)
|
||||
code_records = results.get("code") or []
|
||||
checks = {
|
||||
"mode_is_full_paired_campaign": mode == "both",
|
||||
"exact_pinned_stratified_dataset_manifest": manifest_exact,
|
||||
"all_84_unique_tasks_present": exact_coverage,
|
||||
**arm_checks,
|
||||
"zero_provider_errors": not errors,
|
||||
"every_code_trajectory_used_python_constraint": (
|
||||
len(code_records) == 84
|
||||
and all(record.get("used_python_constraint") for record in code_records)
|
||||
),
|
||||
}
|
||||
return {
|
||||
"status": "complete" if all(checks.values()) else "incomplete",
|
||||
"checks": checks,
|
||||
"provider_errors": errors,
|
||||
"expected_tasks_per_arm": 84,
|
||||
}
|
||||
|
||||
|
||||
def print_table(columns, puzzles):
|
||||
"""打印多列准确率对比表。columns = [(mode, records), ...]，顺序即列顺序。"""
|
||||
accs = {m: sum(r["correct"] for r in recs) / len(recs) for m, recs in columns}
|
||||
header = f"{'题号':<8}{'人数':<6}" + "".join(f"{LABELS[m]:<10}" for m, _ in columns)
|
||||
print("\n" + "=" * 60)
|
||||
print("准确率对比表")
|
||||
print("=" * 60)
|
||||
print(header)
|
||||
print("-" * 60)
|
||||
n = len(puzzles)
|
||||
for i in range(n):
|
||||
row = f"{puzzles[i]['id']:<8}{puzzles[i]['num_people']:<6}"
|
||||
for _, recs in columns:
|
||||
row += f"{('✓' if recs[i]['correct'] else '✗'):<10}"
|
||||
print(row)
|
||||
print("-" * 60)
|
||||
tail = f"{'准确率':<8}{'':<6}" + "".join(
|
||||
f"{accs[m]*100:>6.1f}% " for m, _ in columns)
|
||||
print(tail)
|
||||
print("=" * 60)
|
||||
for m, recs in columns:
|
||||
n_ok = sum(r["correct"] for r in recs)
|
||||
print(f"{LABELS[m]:<6} 准确率: {accs[m]*100:5.1f}% ({n_ok}/{len(recs)})")
|
||||
# 若同时有 solver/code 与 pure，报告提升幅度
|
||||
baseline = next((m for m in ("pure",) if m in accs), None)
|
||||
best = next((m for m in ("solver", "code") if m in accs), None)
|
||||
if baseline and best and best != baseline:
|
||||
print(f"提升({LABELS[best]} - {LABELS[baseline]}): "
|
||||
f"{(accs[best]-accs[baseline])*100:+.1f} 个百分点")
|
||||
|
||||
|
||||
def main():
|
||||
global MODEL
|
||||
ap = argparse.ArgumentParser(
|
||||
description="实验 5-2：对比纯思考 / 代码辅助 / 约束求解 三种模式求解"
|
||||
"「骑士与无赖」逻辑谜题的准确率",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=__doc__)
|
||||
ap.add_argument("--mode", choices=["both", "pure", "code", "solver"],
|
||||
default="both",
|
||||
help="运行模式：both=纯思考+代码辅助(默认)；pure=仅纯思考；"
|
||||
"code=仅代码辅助；solver=离线约束求解基线(无需 API)")
|
||||
ap.add_argument("--model", default=MODEL,
|
||||
help=f"LLM 模型名(默认 {MODEL}；solver 模式忽略)")
|
||||
ap.add_argument(
|
||||
"--provider",
|
||||
choices=["auto", "ollama", "openai", "openrouter", "moonshot", "ark"],
|
||||
default="auto",
|
||||
help="explicit API provider; recorded in the saved evidence",
|
||||
)
|
||||
ap.add_argument("--limit", type=int, default=0,
|
||||
help="只跑前 N 题(0=全部)")
|
||||
ap.add_argument("--min-people", type=int, default=0,
|
||||
help="只跑居民数 >= 该值的谜题(按难度筛选，0=不限)")
|
||||
ap.add_argument("--max-people", type=int, default=0,
|
||||
help="只跑居民数 <= 该值的谜题(按难度筛选，0=不限)")
|
||||
ap.add_argument("--puzzles", default="puzzles.json",
|
||||
help="谜题数据集路径(默认 puzzles.json)")
|
||||
ap.add_argument("--output", default="last_run.json",
|
||||
help="逐题完整记录的输出路径(默认 last_run.json)")
|
||||
ap.add_argument(
|
||||
"--manifest", default=None,
|
||||
help="optional dataset manifest; defaults to the matching .manifest.json",
|
||||
)
|
||||
ap.add_argument(
|
||||
"--resume", action="store_true",
|
||||
help="resume successful per-arm rows from OUTPUT.checkpoint.json",
|
||||
)
|
||||
args = ap.parse_args()
|
||||
MODEL = args.model
|
||||
|
||||
with open(args.puzzles, encoding="utf-8") as f:
|
||||
puzzles = json.load(f)
|
||||
if args.min_people:
|
||||
puzzles = [p for p in puzzles if p["num_people"] >= args.min_people]
|
||||
if args.max_people:
|
||||
puzzles = [p for p in puzzles if p["num_people"] <= args.max_people]
|
||||
if args.limit:
|
||||
puzzles = puzzles[:args.limit]
|
||||
if not puzzles:
|
||||
sys.exit("错误：筛选后没有任何谜题，请放宽 --min-people/--max-people/--limit。")
|
||||
|
||||
# solver 模式完全离线，不需要 API；其余模式需要 OPENAI_API_KEY。
|
||||
llm_modes = {"both": ["pure", "code"], "pure": ["pure"],
|
||||
"code": ["code"], "solver": []}[args.mode]
|
||||
results = {}
|
||||
checkpoint_path = Path(str(args.output) + ".checkpoint.json")
|
||||
resumed_results = {}
|
||||
if args.resume and checkpoint_path.is_file():
|
||||
prior = json.loads(checkpoint_path.read_text(encoding="utf-8"))
|
||||
if prior.get("provider") != args.provider or prior.get("model") != MODEL:
|
||||
raise ValueError("resume rejected: provider/model changed")
|
||||
if prior.get("puzzle_ids") != [puzzle["id"] for puzzle in puzzles]:
|
||||
raise ValueError("resume rejected: selected puzzle set changed")
|
||||
resumed_results = prior.get("results") or {}
|
||||
|
||||
checkpoint_results = {
|
||||
arm: list(records) for arm, records in resumed_results.items()
|
||||
}
|
||||
|
||||
def persist_checkpoint(arm, records):
|
||||
checkpoint_results[arm] = list(records)
|
||||
checkpoint_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
checkpoint_path.write_text(json.dumps({
|
||||
"schema_version": "1.0", "experiment": "5-2",
|
||||
"provider": args.provider, "model": MODEL,
|
||||
"puzzle_ids": [puzzle["id"] for puzzle in puzzles],
|
||||
"results": checkpoint_results,
|
||||
}, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
|
||||
if args.mode == "solver":
|
||||
print(f"离线约束求解基线 题目数：{len(puzzles)}\n")
|
||||
print("== 约束求解(solver，离线) ==")
|
||||
results["solver"] = run_solver(puzzles)
|
||||
else:
|
||||
client, MODEL = build_client_and_model(args.provider)
|
||||
print(f"供应商：{PROVIDER} 模型：{MODEL} 题目数：{len(puzzles)} 模式：{args.mode}\n")
|
||||
for m in llm_modes:
|
||||
print(f"== {LABELS[m]}({m}) ==")
|
||||
results[m] = run_mode(
|
||||
client, puzzles, m,
|
||||
existing=resumed_results.get(m),
|
||||
checkpoint=persist_checkpoint,
|
||||
)
|
||||
print()
|
||||
|
||||
# ---- 准确率对比表(按 pure -> code -> solver 的固定列序) ----
|
||||
columns = [(m, results[m]) for m in ["pure", "code", "solver"] if m in results]
|
||||
print_table(columns, puzzles)
|
||||
|
||||
# ---- 展示一题的约束建模代码与求解结果 ----
|
||||
code_recs = results.get("code")
|
||||
if code_recs:
|
||||
sample = next((r for r in code_recs if r["correct"] and r["codes"]), None)
|
||||
if sample:
|
||||
print("\n" + "=" * 60)
|
||||
print(f"示例：{sample['id']} 的约束建模代码(模型生成)")
|
||||
print("=" * 60)
|
||||
print(sample["codes"][0])
|
||||
print("-- 求解 & 最终答案 --")
|
||||
print(f"预测={sample['pred']} 真值={sample['gold']}")
|
||||
|
||||
# 保存完整记录，便于复盘
|
||||
puzzle_path = Path(args.puzzles)
|
||||
manifest_path = (
|
||||
Path(args.manifest) if args.manifest
|
||||
else puzzle_path.with_name(puzzle_path.stem + ".manifest.json")
|
||||
)
|
||||
manifest = None
|
||||
manifest_sha256 = None
|
||||
if manifest_path.is_file():
|
||||
import hashlib
|
||||
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
|
||||
manifest_sha256 = hashlib.sha256(manifest_path.read_bytes()).hexdigest()
|
||||
payload = dict(
|
||||
schema_version="2.0",
|
||||
experiment="5-2",
|
||||
generated_at_utc=dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
provider=PROVIDER,
|
||||
model=MODEL,
|
||||
mode=args.mode,
|
||||
tasks=len(puzzles),
|
||||
dataset_manifest=manifest,
|
||||
dataset_manifest_sha256=manifest_sha256,
|
||||
)
|
||||
for m, recs in results.items():
|
||||
payload[m] = recs
|
||||
payload[f"{m}_acc"] = sum(r["correct"] for r in recs) / len(recs)
|
||||
if "pure" in results and "code" in results:
|
||||
payload["paired_analysis"] = paired_statistics(results["pure"], results["code"])
|
||||
payload["completion"] = campaign_completion(
|
||||
results, puzzles, manifest, args.mode
|
||||
)
|
||||
payload["official_complete"] = payload["completion"]["status"] == "complete"
|
||||
output_path = Path(args.output)
|
||||
output_path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with output_path.open("w", encoding="utf-8") as f:
|
||||
json.dump(payload, f, ensure_ascii=False, indent=2)
|
||||
print(f"\n完整逐题记录已保存到 {args.output}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,14 @@
|
||||
# 复制本文件为 .env 并填入你的密钥
|
||||
# 必填其一：OpenAI API Key（直连）
|
||||
OPENAI_API_KEY=your-openai-api-key
|
||||
|
||||
# 通用兜底：未配置 OPENAI_API_KEY 时自动改走 OpenRouter（route openai/gpt-4o-mini）。
|
||||
# 仅当把 MODEL 换成 gpt-5.x 这类需组织实名认证的模型、且设置了本 key 时，
|
||||
# 才会优先走 OpenRouter。
|
||||
# OPENROUTER_API_KEY=your-openrouter-api-key
|
||||
|
||||
# 可选：切换到兼容 OpenAI 协议的服务端点
|
||||
# OPENAI_BASE_URL=https://api.openai.com/v1
|
||||
|
||||
# 可选：默认模型(不填则用 gpt-4o-mini)
|
||||
MODEL=gpt-4o-mini
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,412 @@
|
||||
{
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-2",
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"sampling": {
|
||||
"split": "test",
|
||||
"perturbations": [
|
||||
"perturbed_leaf",
|
||||
"perturbed_statement",
|
||||
"reorder_statement",
|
||||
"random_pair",
|
||||
"uncommon_name",
|
||||
"flip_role"
|
||||
],
|
||||
"people": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8
|
||||
],
|
||||
"per_cell": 2,
|
||||
"seed": 512,
|
||||
"cells": 42,
|
||||
"total": 84
|
||||
},
|
||||
"source_files": [
|
||||
{
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"published_rows": 76,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
11
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people3_num100.jsonl",
|
||||
"sha256": "4c3d767065edb62d19728da7a4e4e4c63f550c1719c3600c868f50f48715711e",
|
||||
"published_rows": 93,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people4_num100.jsonl",
|
||||
"sha256": "b3811d1ea4bbdb2ade55dde240361470516e0541c58b5c526609f99ad6a038d7",
|
||||
"published_rows": 94,
|
||||
"sampled_rows": [
|
||||
66,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people5_num100.jsonl",
|
||||
"sha256": "17529110798b0a659286983d0cf4e0d3b5a2b560dfb6ab3b3acb1b61ac9d7da7",
|
||||
"published_rows": 98,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
41
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people6_num100.jsonl",
|
||||
"sha256": "032c0c253bbe574335c741bb5ba7d07b48f622e217dc939f72a633ef9fbe46b0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
17,
|
||||
51
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people7_num100.jsonl",
|
||||
"sha256": "f62b1f0c3c45e3a62660951fc12b9716679f1466bdc313066d85d08d6fe3c767",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
79,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people8_num100.jsonl",
|
||||
"sha256": "01947375b3a16ce4fb340d6f0e1545d5aeadfc6f695c49cf2a6f436fcc5ba099",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
11,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people2_num100.jsonl",
|
||||
"sha256": "18479f9b2c327e9ce51e901479e9e7ffcd4397094e91fdca430871a3fc131fc3",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
94
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people3_num100.jsonl",
|
||||
"sha256": "cb8832b4aae19426e10a039236f0e4fb7dad797598edb3850db8067ff7cda1a6",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
24
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people4_num100.jsonl",
|
||||
"sha256": "ec8821279ff09db70098b97c5b47f452f2b9a6059b3d32ff22c9fce8342e4c30",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
27,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people5_num100.jsonl",
|
||||
"sha256": "791a14c3a7fea7713904786cdbd24ccc94901c0508c1ed3daf5abe3794857ed4",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
95,
|
||||
99
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people6_num100.jsonl",
|
||||
"sha256": "afadf008aa8aba0753bb198baca07ea51ebd9879045b619861225eea56559134",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
47,
|
||||
52
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people7_num100.jsonl",
|
||||
"sha256": "bd3900ae3527887659a1a0da1146780e0a47e4d96ff301e1c2039f35bcfff849",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
15,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people8_num100.jsonl",
|
||||
"sha256": "c3d91ba046bbeb5ec0d64c9147d3d2c76c1f631716da47d32b8379ebfcbe35a0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
86
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people2_num100.jsonl",
|
||||
"sha256": "c116f384f984150969554d783ccf7cecf4c6e56c59d2f5d326bfcfa722bb149c",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
60,
|
||||
76
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people3_num100.jsonl",
|
||||
"sha256": "ff772d57162fb500e608ef28c59eaf57ba1d80f60cf3f73902162c3476f6b0a2",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
64
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people4_num100.jsonl",
|
||||
"sha256": "be12e4b5a38792785264c16c11117f7f10f955517e32c92ea37949017a4da0cb",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people5_num100.jsonl",
|
||||
"sha256": "04a492522cdeb64a960632652b54e3fd27cc505fa8a1c48cf9c666cc8e7d6fb0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
55,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people6_num100.jsonl",
|
||||
"sha256": "4b6131d749c47ce1580d0443e373112cc5f0e20f1b0245455b3df63b106580c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
80,
|
||||
85
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people7_num100.jsonl",
|
||||
"sha256": "d3051b151ef7e5c47a9cd69b09b52f1ec94c7f899821ec685e286e578c7932c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
67,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people8_num100.jsonl",
|
||||
"sha256": "3710c66680d201ebf3d6d243fa9f54ee7e5ce35570ed44f87cd5111b05dc1d9b",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
2,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people2_num100.jsonl",
|
||||
"sha256": "92b9d18d12ea3eaaa0a49daa03f519b5c5f66614bf724e541e9c00e1b9be24ae",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
13,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people3_num100.jsonl",
|
||||
"sha256": "6eb60022244be6fa25d372d1580eb8dddfb06914d7290dbe1d64c9cb57c1f612",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
54,
|
||||
75
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people4_num100.jsonl",
|
||||
"sha256": "1cafedf8ea1c5bf393d2366a6d798e66730059fdd5290c8b2a8cca35de6c4589",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
90
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people5_num100.jsonl",
|
||||
"sha256": "9036535b0ace7b3cfe1b8cf94fbb898b0c07e8d88d23cbc914707ed6d0221ebc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
62,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people6_num100.jsonl",
|
||||
"sha256": "9d3a4bcd28f465b311bd5cdc96958b69529240f05e00a79d80ea7a54b8d53002",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
45
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people7_num100.jsonl",
|
||||
"sha256": "389713889ec22d14bf64d1bb9b846d1fb7a4adba44004b6ce9db1e0ac3d54190",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people8_num100.jsonl",
|
||||
"sha256": "8885fe0d63bfe5910617bd84e6be646f7b62a26978a649b4173889601d0f9229",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
71
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people2_num100.jsonl",
|
||||
"sha256": "25437adfc79438934dbaa113f4a2b06378d3f899a3cc9d8193202a3e0d746f29",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people3_num100.jsonl",
|
||||
"sha256": "087b47837520c8da31d78deac08781c3a62ad7961bae246d68f81843b3795ab0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
40
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people4_num100.jsonl",
|
||||
"sha256": "8e0edc42da3d5739a4e24f82725c6332e71189016391810b5e80d549c71d1004",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
44,
|
||||
57
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people5_num100.jsonl",
|
||||
"sha256": "c1e7ade7a1791ab17a1d951f7548ce8c54767190f0402e4e6785889810694f3f",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
5,
|
||||
47
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people6_num100.jsonl",
|
||||
"sha256": "4046b49fe876c7f4eab6eddc35ff0a9f01ad487faf59e9edb5605a39b3e56530",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
56
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people7_num100.jsonl",
|
||||
"sha256": "752a97113d999fcff5f2910c78fca761133822b99b5ae6ec9070421e0e51b2dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
34,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people8_num100.jsonl",
|
||||
"sha256": "951228b33ff7b4afffcbd247eeb67390d9af5ff2def9f9cdb0f8433564fffc15",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
12,
|
||||
68
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people2_num100.jsonl",
|
||||
"sha256": "2885eb7b356f9ab67ff2ccef9372dcfa7ebcf6ed2b8c5e720db1d74f03840c8d",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
7,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people3_num100.jsonl",
|
||||
"sha256": "31026489710814617c66a6879408accc099c34f395425b41d0bd56743d933461",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people4_num100.jsonl",
|
||||
"sha256": "a5d56cc8724a40fd674dba7730f52ca8dec26966f07ff434302bd62b1260a458",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people5_num100.jsonl",
|
||||
"sha256": "1429f88b80794e12f731bcd9cd2502384a3c4ac1a3292010103baccd86e93ae7",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
73
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people6_num100.jsonl",
|
||||
"sha256": "ccfee7ea7e2f24376e88193a979a20ac6410ec2458c534af7e4daedab4ff47ff",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
39,
|
||||
62
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people7_num100.jsonl",
|
||||
"sha256": "b5720e5a434747f323f90086ac6bae7ba33d1b6ba8d617424f599bfabdbcdeca",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
1,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people8_num100.jsonl",
|
||||
"sha256": "d668973f128b119ad065e0b58f33099516dbd801f0e7957c00e876115e8119dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
96
|
||||
]
|
||||
}
|
||||
],
|
||||
"label_validation": "all rows independently solved with python-constraint"
|
||||
}
|
||||
@@ -0,0 +1,505 @@
|
||||
[
|
||||
{
|
||||
"id": "kk01",
|
||||
"num_people": 2,
|
||||
"names": [
|
||||
"A",
|
||||
"B"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 是无赖。",
|
||||
"B": "我们两人都不是骑士。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"is",
|
||||
"B",
|
||||
"knave"
|
||||
],
|
||||
"B": [
|
||||
"and",
|
||||
[
|
||||
"is",
|
||||
"A",
|
||||
"knave"
|
||||
],
|
||||
[
|
||||
"is",
|
||||
"B",
|
||||
"knave"
|
||||
]
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 2 位居民：A, B。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 是无赖。」\nB: 「我们两人都不是骑士。」",
|
||||
"solution": {
|
||||
"A": "knight",
|
||||
"B": "knave"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk02",
|
||||
"num_people": 2,
|
||||
"names": [
|
||||
"A",
|
||||
"B"
|
||||
],
|
||||
"statements": {
|
||||
"A": "我和 B 是同一类人（要么都是骑士，要么都是无赖）。",
|
||||
"B": "我和 A 是不同类人。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"same",
|
||||
"A",
|
||||
"B"
|
||||
],
|
||||
"B": [
|
||||
"diff",
|
||||
"A",
|
||||
"B"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 2 位居民：A, B。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「我和 B 是同一类人（要么都是骑士，要么都是无赖）。」\nB: 「我和 A 是不同类人。」",
|
||||
"solution": {
|
||||
"A": "knave",
|
||||
"B": "knight"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk03",
|
||||
"num_people": 2,
|
||||
"names": [
|
||||
"A",
|
||||
"B"
|
||||
],
|
||||
"statements": {
|
||||
"A": "我们当中至少有一个骑士。",
|
||||
"B": "A 是无赖。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"count",
|
||||
"knight",
|
||||
">=",
|
||||
1
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"A",
|
||||
"knave"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 2 位居民：A, B。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「我们当中至少有一个骑士。」\nB: 「A 是无赖。」",
|
||||
"solution": {
|
||||
"A": "knight",
|
||||
"B": "knave"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk04",
|
||||
"num_people": 3,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 是无赖。",
|
||||
"B": "C 是无赖。",
|
||||
"C": "A 和 B 都是无赖。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"is",
|
||||
"B",
|
||||
"knave"
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"C",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"and",
|
||||
[
|
||||
"is",
|
||||
"A",
|
||||
"knave"
|
||||
],
|
||||
[
|
||||
"is",
|
||||
"B",
|
||||
"knave"
|
||||
]
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 3 位居民：A, B, C。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 是无赖。」\nB: 「C 是无赖。」\nC: 「A 和 B 都是无赖。」",
|
||||
"solution": {
|
||||
"A": "knave",
|
||||
"B": "knight",
|
||||
"C": "knave"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk05",
|
||||
"num_people": 3,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 是骑士。",
|
||||
"B": "C 是无赖。",
|
||||
"C": "A 和 B 是同一类人。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"is",
|
||||
"B",
|
||||
"knight"
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"C",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"same",
|
||||
"A",
|
||||
"B"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 3 位居民：A, B, C。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 是骑士。」\nB: 「C 是无赖。」\nC: 「A 和 B 是同一类人。」",
|
||||
"solution": {
|
||||
"A": "knave",
|
||||
"B": "knave",
|
||||
"C": "knight"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk06",
|
||||
"num_people": 3,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 和 C 是同一类人。",
|
||||
"B": "A 是无赖。",
|
||||
"C": "我和 A 是同一类人。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"same",
|
||||
"B",
|
||||
"C"
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"A",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"same",
|
||||
"C",
|
||||
"A"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 3 位居民：A, B, C。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 和 C 是同一类人。」\nB: 「A 是无赖。」\nC: 「我和 A 是同一类人。」",
|
||||
"solution": {
|
||||
"A": "knight",
|
||||
"B": "knave",
|
||||
"C": "knave"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk07",
|
||||
"num_people": 3,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C"
|
||||
],
|
||||
"statements": {
|
||||
"A": "我是无赖，或者 B 是骑士。",
|
||||
"B": "A 是骑士。",
|
||||
"C": "B 是无赖。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"or",
|
||||
[
|
||||
"is",
|
||||
"A",
|
||||
"knave"
|
||||
],
|
||||
[
|
||||
"is",
|
||||
"B",
|
||||
"knight"
|
||||
]
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"A",
|
||||
"knight"
|
||||
],
|
||||
"C": [
|
||||
"is",
|
||||
"B",
|
||||
"knave"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 3 位居民：A, B, C。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「我是无赖，或者 B 是骑士。」\nB: 「A 是骑士。」\nC: 「B 是无赖。」",
|
||||
"solution": {
|
||||
"A": "knight",
|
||||
"B": "knight",
|
||||
"C": "knave"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk08",
|
||||
"num_people": 4,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C",
|
||||
"D"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 和 D 是同一类人。",
|
||||
"B": "C 是无赖。",
|
||||
"C": "D 是骑士。",
|
||||
"D": "B 和 C 是不同类人。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"same",
|
||||
"B",
|
||||
"D"
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"C",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"is",
|
||||
"D",
|
||||
"knight"
|
||||
],
|
||||
"D": [
|
||||
"diff",
|
||||
"B",
|
||||
"C"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 4 位居民：A, B, C, D。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 和 D 是同一类人。」\nB: 「C 是无赖。」\nC: 「D 是骑士。」\nD: 「B 和 C 是不同类人。」",
|
||||
"solution": {
|
||||
"A": "knave",
|
||||
"B": "knave",
|
||||
"C": "knight",
|
||||
"D": "knight"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk09",
|
||||
"num_people": 4,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C",
|
||||
"D"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 是骑士。",
|
||||
"B": "C 是无赖。",
|
||||
"C": "D 是骑士。",
|
||||
"D": "A 和 B 不是同一类人。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"is",
|
||||
"B",
|
||||
"knight"
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"C",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"is",
|
||||
"D",
|
||||
"knight"
|
||||
],
|
||||
"D": [
|
||||
"diff",
|
||||
"A",
|
||||
"B"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 4 位居民：A, B, C, D。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 是骑士。」\nB: 「C 是无赖。」\nC: 「D 是骑士。」\nD: 「A 和 B 不是同一类人。」",
|
||||
"solution": {
|
||||
"A": "knight",
|
||||
"B": "knight",
|
||||
"C": "knave",
|
||||
"D": "knave"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk10",
|
||||
"num_people": 4,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C",
|
||||
"D"
|
||||
],
|
||||
"statements": {
|
||||
"A": "我们四人当中至少有三个无赖。",
|
||||
"B": "A 是无赖。",
|
||||
"C": "B 是骑士。",
|
||||
"D": "C 是无赖。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"count",
|
||||
"knave",
|
||||
">=",
|
||||
3
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"A",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"is",
|
||||
"B",
|
||||
"knight"
|
||||
],
|
||||
"D": [
|
||||
"is",
|
||||
"C",
|
||||
"knave"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 4 位居民：A, B, C, D。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「我们四人当中至少有三个无赖。」\nB: 「A 是无赖。」\nC: 「B 是骑士。」\nD: 「C 是无赖。」",
|
||||
"solution": {
|
||||
"A": "knave",
|
||||
"B": "knight",
|
||||
"C": "knight",
|
||||
"D": "knave"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk11",
|
||||
"num_people": 5,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C",
|
||||
"D",
|
||||
"E"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 是骑士。",
|
||||
"B": "C 是无赖。",
|
||||
"C": "D 是骑士。",
|
||||
"D": "E 是无赖。",
|
||||
"E": "我们五人当中至少有两个骑士。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"is",
|
||||
"B",
|
||||
"knight"
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"C",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"is",
|
||||
"D",
|
||||
"knight"
|
||||
],
|
||||
"D": [
|
||||
"is",
|
||||
"E",
|
||||
"knave"
|
||||
],
|
||||
"E": [
|
||||
"count",
|
||||
"knight",
|
||||
">=",
|
||||
2
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 5 位居民：A, B, C, D, E。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 是骑士。」\nB: 「C 是无赖。」\nC: 「D 是骑士。」\nD: 「E 是无赖。」\nE: 「我们五人当中至少有两个骑士。」",
|
||||
"solution": {
|
||||
"A": "knight",
|
||||
"B": "knight",
|
||||
"C": "knave",
|
||||
"D": "knave",
|
||||
"E": "knight"
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": "kk12",
|
||||
"num_people": 5,
|
||||
"names": [
|
||||
"A",
|
||||
"B",
|
||||
"C",
|
||||
"D",
|
||||
"E"
|
||||
],
|
||||
"statements": {
|
||||
"A": "B 是骑士。",
|
||||
"B": "C 是无赖。",
|
||||
"C": "D 是无赖。",
|
||||
"D": "E 是骑士。",
|
||||
"E": "A 和 C 是同一类人。"
|
||||
},
|
||||
"statements_struct": {
|
||||
"A": [
|
||||
"is",
|
||||
"B",
|
||||
"knight"
|
||||
],
|
||||
"B": [
|
||||
"is",
|
||||
"C",
|
||||
"knave"
|
||||
],
|
||||
"C": [
|
||||
"is",
|
||||
"D",
|
||||
"knave"
|
||||
],
|
||||
"D": [
|
||||
"is",
|
||||
"E",
|
||||
"knight"
|
||||
],
|
||||
"E": [
|
||||
"same",
|
||||
"A",
|
||||
"C"
|
||||
]
|
||||
},
|
||||
"description": "这座岛上有 5 位居民：A, B, C, D, E。每位居民要么是永远说真话的骑士(knight)，要么是永远说假话的无赖(knave)。他们各自说了如下的话：\nA: 「B 是骑士。」\nB: 「C 是无赖。」\nC: 「D 是无赖。」\nD: 「E 是骑士。」\nE: 「A 和 C 是同一类人。」",
|
||||
"solution": {
|
||||
"A": "knave",
|
||||
"B": "knave",
|
||||
"C": "knight",
|
||||
"D": "knave",
|
||||
"E": "knave"
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,2 @@
|
||||
openai>=1.30.0
|
||||
python-constraint2>=2.0.2
|
||||
@@ -0,0 +1,52 @@
|
||||
"""
|
||||
极简 Code Interpreter 沙箱：在子进程中执行模型生成的 Python 代码。
|
||||
|
||||
- 用独立子进程运行，避免污染主进程、并可强制超时。
|
||||
- 子进程使用与主程序相同的解释器(sys.executable)，因此已预装 python-constraint。
|
||||
- 捕获 stdout / stderr 一并返回给模型，让它能看到求解结果或报错信息。
|
||||
"""
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import os
|
||||
|
||||
|
||||
def run_python(code: str, timeout: int = 20) -> str:
|
||||
"""在子进程沙箱中执行 code，返回合并后的 stdout/stderr 文本。"""
|
||||
with tempfile.NamedTemporaryFile("w", suffix=".py", delete=False,
|
||||
encoding="utf-8") as f:
|
||||
f.write(code)
|
||||
path = f.name
|
||||
try:
|
||||
proc = subprocess.run(
|
||||
[sys.executable, path],
|
||||
capture_output=True, text=True, timeout=timeout,
|
||||
)
|
||||
out = proc.stdout
|
||||
if proc.stderr.strip():
|
||||
out += "\n[stderr]\n" + proc.stderr
|
||||
if not out.strip():
|
||||
out = "(代码已执行，但没有任何输出。记得用 print() 打印结果。)"
|
||||
return out.strip()
|
||||
except subprocess.TimeoutExpired:
|
||||
return f"[错误] 代码执行超时（超过 {timeout} 秒）。"
|
||||
finally:
|
||||
os.unlink(path)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 自测：用 python-constraint 求解一个最简单的 K&K 谜题
|
||||
demo = """
|
||||
from constraint import Problem
|
||||
p = Problem()
|
||||
# True=骑士(说真话), False=无赖(说假话)
|
||||
p.addVariable('A', [True, False])
|
||||
p.addVariable('B', [True, False])
|
||||
# A 说"B 是无赖"：A 的真值 == (B 是无赖) 即 A == (not B)
|
||||
p.addConstraint(lambda a, b: a == (not b), ['A', 'B'])
|
||||
# B 说"我们都不是骑士"：B == (not A and not B)
|
||||
p.addConstraint(lambda a, b: b == ((not a) and (not b)), ['A', 'B'])
|
||||
for s in p.getSolutions():
|
||||
print({k: 'knight' if v else 'knave' for k, v in s.items()})
|
||||
"""
|
||||
print(run_python(demo))
|
||||
@@ -0,0 +1,27 @@
|
||||
from build_hf_puzzles import convert_expression, convert_row
|
||||
|
||||
|
||||
def test_convert_dataset_expression_and_label():
|
||||
row = {
|
||||
"quiz": "A says B is not lying. B says B is truthful iff A is lying.",
|
||||
"names": ["A", "B"],
|
||||
"solution": [False, False],
|
||||
"statements": "(('not', ('lying', 1)), ('<=>', ('telling-truth', 1), ('lying', 0)))",
|
||||
"index": 7,
|
||||
}
|
||||
puzzle = convert_row(
|
||||
row,
|
||||
perturbation="perturbed_leaf",
|
||||
people=2,
|
||||
source_path="test/perturbed_leaf/people2_num100.jsonl",
|
||||
source_sha256="abc",
|
||||
source_row=7,
|
||||
)
|
||||
assert puzzle["solution"] == {"A": "knave", "B": "knave"}
|
||||
assert puzzle["source"]["dataset_index"] == 7
|
||||
|
||||
|
||||
def test_convert_implication_and_biconditional():
|
||||
names = ["A", "B"]
|
||||
assert convert_expression(("->", ("lying", 0), ("telling-truth", 1)), names)[0] == "implies"
|
||||
assert convert_expression(("<=>", ("lying", 0), ("telling-truth", 1)), names)[0] == "iff"
|
||||
@@ -0,0 +1,32 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Ensure demo module can be resolved regardless of working directory
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from demo import parse_answer
|
||||
|
||||
|
||||
def test_parse_answer_supports_boolean_json_values():
|
||||
"""Verify parse_answer maps JSON boolean true/false to knight/knave.
|
||||
|
||||
Contract: LLMs outputting JSON solutions with boolean values like {"A": true, "B": false}
|
||||
must be normalized to {"A": "knight", "B": "knave"} instead of returning None.
|
||||
Locks out KeyError or unparsed boolean JSON answers.
|
||||
"""
|
||||
text = 'The solution is {"A": true, "B": false}'
|
||||
ans = parse_answer(text, ["A", "B"])
|
||||
assert ans == {"A": "knight", "B": "knave"}
|
||||
|
||||
|
||||
def test_parse_answer_supports_numeric_and_str_boolean():
|
||||
"""Verify parse_answer maps numeric 1/0 and string "true"/"false" booleans to knight/knave.
|
||||
|
||||
Contract: 1 and "true" map to "knight"; 0 and "false" map to "knave".
|
||||
Locks out unparsed numeric or string boolean representation in model output.
|
||||
"""
|
||||
text1 = '{"A": 1, "B": 0}'
|
||||
assert parse_answer(text1, ["A", "B"]) == {"A": "knight", "B": "knave"}
|
||||
|
||||
text2 = '{"A": "true", "B": "false"}'
|
||||
assert parse_answer(text2, ["A", "B"]) == {"A": "knight", "B": "knave"}
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Regression: one-person puzzles must not IndexError in _random_stmt."""
|
||||
import random
|
||||
import sys
|
||||
import types
|
||||
|
||||
sys.modules.setdefault("constraint", types.ModuleType("constraint"))
|
||||
sys.modules["constraint"].Problem = object
|
||||
cs = types.ModuleType("csp_solver")
|
||||
cs.render_nl = lambda *a, **k: ""
|
||||
cs.solve = lambda *a, **k: [{}]
|
||||
cs.solve_labeled = lambda *a, **k: {}
|
||||
sys.modules["csp_solver"] = cs
|
||||
|
||||
from build_puzzles import _random_stmt # noqa: E402
|
||||
|
||||
|
||||
def test_random_stmt_one_person_returns_count():
|
||||
rng = random.Random(0)
|
||||
for _ in range(20):
|
||||
stmt = _random_stmt("A", ["A"], rng)
|
||||
assert stmt[0] == "count"
|
||||
assert stmt[1] in ("knight", "knave")
|
||||
@@ -0,0 +1,30 @@
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# Ensure csp_solver module can be resolved regardless of working directory
|
||||
sys.path.insert(0, str(Path(__file__).parent))
|
||||
|
||||
from csp_solver import solve, solve_labeled
|
||||
|
||||
|
||||
def test_csp_solver_handles_silent_resident():
|
||||
"""Verify solve handles residents in names that make no statements.
|
||||
|
||||
Contract: In Knights and Knaves puzzles, residents listed in `names` may be
|
||||
silent (spoken about by others without making statements themselves).
|
||||
`solve` must skip adding speaker-statement constraints for silent residents
|
||||
rather than raising KeyError.
|
||||
"""
|
||||
names = ["A", "B"]
|
||||
# A speaks about B ("B is a knave"), but B makes no statement.
|
||||
structs = {"A": ["is", "B", "knave"]}
|
||||
|
||||
solutions = solve(names, structs)
|
||||
assert len(solutions) == 2
|
||||
# If A is knight (True), B must be knave (False); if A is knave (False), B must be knight (True).
|
||||
assert {"A": True, "B": False} in solutions
|
||||
assert {"A": False, "B": True} in solutions
|
||||
|
||||
labeled = solve_labeled(names, structs)
|
||||
assert {"A": "knight", "B": "knave"} in labeled
|
||||
assert {"A": "knave", "B": "knight"} in labeled
|
||||
@@ -0,0 +1,42 @@
|
||||
from demo import campaign_completion, paired_statistics
|
||||
|
||||
|
||||
def test_paired_statistics_detects_clear_code_gain():
|
||||
pure = [{"id": str(i), "correct": i < 2} for i in range(20)]
|
||||
code = [
|
||||
{"id": str(i), "correct": i < 19, "used_python_constraint": True}
|
||||
for i in range(20)
|
||||
]
|
||||
result = paired_statistics(pure, code)
|
||||
assert result["code_accuracy"] == 0.95
|
||||
assert result["acceptance"]["code_accuracy_over_90_percent"] is True
|
||||
assert result["acceptance"]["code_significantly_higher_than_pure"] is True
|
||||
|
||||
|
||||
def test_campaign_completion_requires_both_full_real_arms():
|
||||
puzzles = [{"id": f"p-{i}"} for i in range(84)]
|
||||
pure = [
|
||||
{"id": p["id"], "provider_receipts": [{"response_id": "r"}],
|
||||
"provider_error": None}
|
||||
for p in puzzles
|
||||
]
|
||||
code = [
|
||||
{"id": p["id"], "provider_receipts": [{"response_id": "r"}],
|
||||
"provider_error": None, "used_python_constraint": True}
|
||||
for p in puzzles
|
||||
]
|
||||
manifest = {
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"sampling": {"total": 84, "cells": 42, "per_cell": 2},
|
||||
"source_files": [{} for _ in range(42)],
|
||||
"label_validation": "all rows independently solved with python-constraint",
|
||||
}
|
||||
result = campaign_completion(
|
||||
{"pure": pure, "code": code}, puzzles, manifest, "both"
|
||||
)
|
||||
assert result["status"] == "complete"
|
||||
code[-1]["used_python_constraint"] = False
|
||||
assert campaign_completion(
|
||||
{"pure": pure, "code": code}, puzzles, manifest, "both"
|
||||
)["status"] == "incomplete"
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because one or more lines are too long
+12984
File diff suppressed because one or more lines are too long
@@ -0,0 +1,594 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "5-2",
|
||||
"generated_at_utc": "2026-07-29T18:26:39.099268+00:00",
|
||||
"provider": "ark",
|
||||
"model": "doubao-seed-1-6-flash-250615",
|
||||
"mode": "both",
|
||||
"tasks": 1,
|
||||
"dataset_manifest": {
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-2",
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"sampling": {
|
||||
"split": "test",
|
||||
"perturbations": [
|
||||
"perturbed_leaf",
|
||||
"perturbed_statement",
|
||||
"reorder_statement",
|
||||
"random_pair",
|
||||
"uncommon_name",
|
||||
"flip_role"
|
||||
],
|
||||
"people": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8
|
||||
],
|
||||
"per_cell": 2,
|
||||
"seed": 512,
|
||||
"cells": 42,
|
||||
"total": 84
|
||||
},
|
||||
"source_files": [
|
||||
{
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"published_rows": 76,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
11
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people3_num100.jsonl",
|
||||
"sha256": "4c3d767065edb62d19728da7a4e4e4c63f550c1719c3600c868f50f48715711e",
|
||||
"published_rows": 93,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people4_num100.jsonl",
|
||||
"sha256": "b3811d1ea4bbdb2ade55dde240361470516e0541c58b5c526609f99ad6a038d7",
|
||||
"published_rows": 94,
|
||||
"sampled_rows": [
|
||||
66,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people5_num100.jsonl",
|
||||
"sha256": "17529110798b0a659286983d0cf4e0d3b5a2b560dfb6ab3b3acb1b61ac9d7da7",
|
||||
"published_rows": 98,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
41
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people6_num100.jsonl",
|
||||
"sha256": "032c0c253bbe574335c741bb5ba7d07b48f622e217dc939f72a633ef9fbe46b0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
17,
|
||||
51
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people7_num100.jsonl",
|
||||
"sha256": "f62b1f0c3c45e3a62660951fc12b9716679f1466bdc313066d85d08d6fe3c767",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
79,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people8_num100.jsonl",
|
||||
"sha256": "01947375b3a16ce4fb340d6f0e1545d5aeadfc6f695c49cf2a6f436fcc5ba099",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
11,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people2_num100.jsonl",
|
||||
"sha256": "18479f9b2c327e9ce51e901479e9e7ffcd4397094e91fdca430871a3fc131fc3",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
94
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people3_num100.jsonl",
|
||||
"sha256": "cb8832b4aae19426e10a039236f0e4fb7dad797598edb3850db8067ff7cda1a6",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
24
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people4_num100.jsonl",
|
||||
"sha256": "ec8821279ff09db70098b97c5b47f452f2b9a6059b3d32ff22c9fce8342e4c30",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
27,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people5_num100.jsonl",
|
||||
"sha256": "791a14c3a7fea7713904786cdbd24ccc94901c0508c1ed3daf5abe3794857ed4",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
95,
|
||||
99
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people6_num100.jsonl",
|
||||
"sha256": "afadf008aa8aba0753bb198baca07ea51ebd9879045b619861225eea56559134",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
47,
|
||||
52
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people7_num100.jsonl",
|
||||
"sha256": "bd3900ae3527887659a1a0da1146780e0a47e4d96ff301e1c2039f35bcfff849",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
15,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people8_num100.jsonl",
|
||||
"sha256": "c3d91ba046bbeb5ec0d64c9147d3d2c76c1f631716da47d32b8379ebfcbe35a0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
86
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people2_num100.jsonl",
|
||||
"sha256": "c116f384f984150969554d783ccf7cecf4c6e56c59d2f5d326bfcfa722bb149c",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
60,
|
||||
76
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people3_num100.jsonl",
|
||||
"sha256": "ff772d57162fb500e608ef28c59eaf57ba1d80f60cf3f73902162c3476f6b0a2",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
64
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people4_num100.jsonl",
|
||||
"sha256": "be12e4b5a38792785264c16c11117f7f10f955517e32c92ea37949017a4da0cb",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people5_num100.jsonl",
|
||||
"sha256": "04a492522cdeb64a960632652b54e3fd27cc505fa8a1c48cf9c666cc8e7d6fb0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
55,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people6_num100.jsonl",
|
||||
"sha256": "4b6131d749c47ce1580d0443e373112cc5f0e20f1b0245455b3df63b106580c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
80,
|
||||
85
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people7_num100.jsonl",
|
||||
"sha256": "d3051b151ef7e5c47a9cd69b09b52f1ec94c7f899821ec685e286e578c7932c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
67,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people8_num100.jsonl",
|
||||
"sha256": "3710c66680d201ebf3d6d243fa9f54ee7e5ce35570ed44f87cd5111b05dc1d9b",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
2,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people2_num100.jsonl",
|
||||
"sha256": "92b9d18d12ea3eaaa0a49daa03f519b5c5f66614bf724e541e9c00e1b9be24ae",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
13,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people3_num100.jsonl",
|
||||
"sha256": "6eb60022244be6fa25d372d1580eb8dddfb06914d7290dbe1d64c9cb57c1f612",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
54,
|
||||
75
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people4_num100.jsonl",
|
||||
"sha256": "1cafedf8ea1c5bf393d2366a6d798e66730059fdd5290c8b2a8cca35de6c4589",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
90
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people5_num100.jsonl",
|
||||
"sha256": "9036535b0ace7b3cfe1b8cf94fbb898b0c07e8d88d23cbc914707ed6d0221ebc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
62,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people6_num100.jsonl",
|
||||
"sha256": "9d3a4bcd28f465b311bd5cdc96958b69529240f05e00a79d80ea7a54b8d53002",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
45
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people7_num100.jsonl",
|
||||
"sha256": "389713889ec22d14bf64d1bb9b846d1fb7a4adba44004b6ce9db1e0ac3d54190",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people8_num100.jsonl",
|
||||
"sha256": "8885fe0d63bfe5910617bd84e6be646f7b62a26978a649b4173889601d0f9229",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
71
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people2_num100.jsonl",
|
||||
"sha256": "25437adfc79438934dbaa113f4a2b06378d3f899a3cc9d8193202a3e0d746f29",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people3_num100.jsonl",
|
||||
"sha256": "087b47837520c8da31d78deac08781c3a62ad7961bae246d68f81843b3795ab0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
40
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people4_num100.jsonl",
|
||||
"sha256": "8e0edc42da3d5739a4e24f82725c6332e71189016391810b5e80d549c71d1004",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
44,
|
||||
57
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people5_num100.jsonl",
|
||||
"sha256": "c1e7ade7a1791ab17a1d951f7548ce8c54767190f0402e4e6785889810694f3f",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
5,
|
||||
47
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people6_num100.jsonl",
|
||||
"sha256": "4046b49fe876c7f4eab6eddc35ff0a9f01ad487faf59e9edb5605a39b3e56530",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
56
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people7_num100.jsonl",
|
||||
"sha256": "752a97113d999fcff5f2910c78fca761133822b99b5ae6ec9070421e0e51b2dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
34,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people8_num100.jsonl",
|
||||
"sha256": "951228b33ff7b4afffcbd247eeb67390d9af5ff2def9f9cdb0f8433564fffc15",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
12,
|
||||
68
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people2_num100.jsonl",
|
||||
"sha256": "2885eb7b356f9ab67ff2ccef9372dcfa7ebcf6ed2b8c5e720db1d74f03840c8d",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
7,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people3_num100.jsonl",
|
||||
"sha256": "31026489710814617c66a6879408accc099c34f395425b41d0bd56743d933461",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people4_num100.jsonl",
|
||||
"sha256": "a5d56cc8724a40fd674dba7730f52ca8dec26966f07ff434302bd62b1260a458",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people5_num100.jsonl",
|
||||
"sha256": "1429f88b80794e12f731bcd9cd2502384a3c4ac1a3292010103baccd86e93ae7",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
73
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people6_num100.jsonl",
|
||||
"sha256": "ccfee7ea7e2f24376e88193a979a20ac6410ec2458c534af7e4daedab4ff47ff",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
39,
|
||||
62
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people7_num100.jsonl",
|
||||
"sha256": "b5720e5a434747f323f90086ac6bae7ba33d1b6ba8d617424f599bfabdbcdeca",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
1,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people8_num100.jsonl",
|
||||
"sha256": "d668973f128b119ad065e0b58f33099516dbd801f0e7957c00e876115e8119dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
96
|
||||
]
|
||||
}
|
||||
],
|
||||
"label_validation": "all rows independently solved with python-constraint"
|
||||
},
|
||||
"dataset_manifest_sha256": "2e55a7d8b07e55b035a8129889b11dff186589bcad97c02c0ce553f38249212f",
|
||||
"pure": [
|
||||
{
|
||||
"id": "perturbed_leaf-p2-r006",
|
||||
"num": 2,
|
||||
"pred": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"gold": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"correct": true,
|
||||
"source": {
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"config": "test",
|
||||
"split": "perturbed_leaf",
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"file_sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"row": 6,
|
||||
"dataset_index": 9
|
||||
},
|
||||
"codes": [],
|
||||
"text": "### Step 1: Analyze James's statement\nJames says, \"Sophia is not a knight.\" Let's denote \\( J \\) as \"James is a knight\" and \\( S \\) as \"Sophia is a knight\". James's statement is \\( \\neg S \\).\n\n### Step 2: Analyze Sophia's statement\nSophia says, \"James is a knight if and only if James is a knave.\" The \"if and only if\" (iff) statement \\( J \\leftrightarrow \\neg J \\) is a contradiction because a statement cannot be both true and false simultaneously.\n\n### Step 3: Assume James is a knight\nIf \\( J \\) is true (James is a knight), then James's statement \\( \\neg S \\) is true, so \\( S \\) is false (Sophia is a knave). Now, check Sophia's statement: \\( J \\leftrightarrow \\neg J \\). Since \\( J \\) is true and \\( \\neg J \\) is false, \\( J \\leftrightarrow \\neg J \\) is false. Sophia is a knave, so she lies, which is consistent with her false statement.\n\n### Step 4: Assume James is a knave\nIf \\( J \\) is false (James is a knave), then James's statement \\( \\neg S \\) is a lie, so \\( S \\) is true (Sophia is a knight). Now, check Sophia's statement: \\( J \\leftrightarrow \\neg J \\). Since \\( J \\) is false and \\( \\neg J \\) is true, \\( J \\leftrightarrow \\neg J \\) is false. But Sophia is a knight, so she must tell the truth. This is a contradiction because her statement is false.\n\nThus, the only consistent case is James is a knight and Sophia is a knave.\n\n{\"James\": \"knight\", \"Sophia\": \"knave\"}",
|
||||
"used_python_constraint": false,
|
||||
"duration_s": 9.42,
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "021785349572412c2531bd3ff98f8d456851518a83e6729925db2",
|
||||
"response_model": "doubao-seed-1-6-flash-250615",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 237,
|
||||
"completion_tokens": 1750,
|
||||
"total_tokens": 1987,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"pure_acc": 1.0,
|
||||
"code": [
|
||||
{
|
||||
"id": "perturbed_leaf-p2-r006",
|
||||
"num": 2,
|
||||
"pred": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"gold": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"correct": true,
|
||||
"source": {
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"config": "test",
|
||||
"split": "perturbed_leaf",
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"file_sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"row": 6,
|
||||
"dataset_index": 9
|
||||
},
|
||||
"codes": [
|
||||
"",
|
||||
"from constraint import Problem\n\np = Problem()\n# 定义变量，True表示骑士，False表示无赖\np.addVariable('James', [True, False])\np.addVariable('Sophia', [True, False])\n\n# James说\"Sophia不是骑士\"，所以James == (Sophia是假)，即James == (not Sophia)\np.addConstraint(lambda james, sophia: james == (not sophia), ['James', 'Sophia'])\n\n# Sophia说\"James是骑士当且仅当James是无赖\"，即Sophia的话等价于James == not James，而James == not James恒假，所以Sophia == False\np.addConstraint(lambda sophia: sophia == False, ['Sophia'])\n\n# 寻找所有解\nsolutions = p.getSolutions()\n\n# 转换结果并打印\nfor sol in solutions:\n result = {}\n for name, val in sol.items():\n result[name] = 'knight' if val else 'knave'\n print(result)",
|
||||
"from constraint import Problem\n\np = Problem()\n# 定义变量，True表示骑士，False表示无赖\np.addVariable('James', [True, False])\np.addVariable('Sophia', [True, False])\n\n# James说\"Sophia不是骑士\"，所以James == (Sophia是假)，即James == (not Sophia)\np.addConstraint(lambda james, sophia: james == (not sophia), ['James', 'Sophia'])\n\n# Sophia说\"James是骑士当且仅当James是无赖\"，即Sophia的话等价于James == not James，而James == not James恒假，所以Sophia == False\np.addConstraint(lambda sophia: sophia == False, ['Sophia'])\n\n# 寻找所有解\nsolutions = p.getSolutions()\n\n# 转换结果并打印\nresults = []\nfor sol in solutions:\n result = {}\n for name, val in sol.items():\n result[name] = 'knight' if val else 'knave'\n results.append(result)\n\n# 这里直接输出最终结果\nprint(results)",
|
||||
"from constraint import Problem\n\np = Problem()\n# 定义变量，True表示骑士，False表示无赖\np.addVariable('James', [True, False])\np.addVariable('Sophia', [True, False])\n\n# James说\"Sophia不是骑士\"，所以James == (Sophia是假)，即James == (not Sophia)\np.addConstraint(lambda james, sophia: james == (not sophia), ['James', 'Sophia'])\n\n# Sophia说\"James是骑士当且仅当James是无赖\"，即Sophia的话等价于James == not James，而James == not James恒假，所以Sophia == False\np.addConstraint(lambda sophia: sophia == False, ['Sophia'])\n\n# 寻找所有解\nsolutions = p.getSolutions()\n\n# 转换结果并输出最终的JSON\nfinal_result = {} \nfor sol in solutions:\n for name, val in sol.items():\n final_result[name] = 'knight' if val else 'knave'\n\nprint(final_result)"
|
||||
],
|
||||
"text": "{\"James\": \"knight\", \"Sophia\": \"knave\"}",
|
||||
"used_python_constraint": true,
|
||||
"duration_s": 17.563,
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "021785349581717c2531bd3ff98f8d456851518a83e672947b029",
|
||||
"response_model": "doubao-seed-1-6-flash-250615",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 1124,
|
||||
"completion_tokens": 437,
|
||||
"total_tokens": 1561,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 1
|
||||
},
|
||||
{
|
||||
"turn": 2,
|
||||
"response_id": "021785349588314c2531bd3ff98f8d456851518a83e672919a4ec",
|
||||
"response_model": "doubao-seed-1-6-flash-250615",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 1453,
|
||||
"completion_tokens": 871,
|
||||
"total_tokens": 2324,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 1
|
||||
},
|
||||
{
|
||||
"turn": 3,
|
||||
"response_id": "021785349593459c2531bd3ff98f8d456851518a83e6729b3f947",
|
||||
"response_model": "doubao-seed-1-6-flash-250615",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 1747,
|
||||
"completion_tokens": 345,
|
||||
"total_tokens": 2092,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 1
|
||||
},
|
||||
{
|
||||
"turn": 4,
|
||||
"response_id": "021785349596925c2531bd3ff98f8d456851518a83e672961e3df",
|
||||
"response_model": "doubao-seed-1-6-flash-250615",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 2062,
|
||||
"completion_tokens": 322,
|
||||
"total_tokens": 2384,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 1
|
||||
},
|
||||
{
|
||||
"turn": 5,
|
||||
"response_id": "021785349598681c2531bd3ff98f8d456851518a83e67292205e8",
|
||||
"response_model": "doubao-seed-1-6-flash-250615",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 2361,
|
||||
"completion_tokens": 70,
|
||||
"total_tokens": 2431,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"code_acc": 1.0,
|
||||
"paired_analysis": {
|
||||
"test": "two-sided exact McNemar/binomial test on discordant pairs",
|
||||
"n": 1,
|
||||
"contingency": {
|
||||
"pure_only": 0,
|
||||
"code_only": 0,
|
||||
"discordant": 0
|
||||
},
|
||||
"pure_accuracy": 1.0,
|
||||
"code_accuracy": 1.0,
|
||||
"accuracy_delta": 0.0,
|
||||
"code_accuracy_wilson_95": [
|
||||
0.20654931437723745,
|
||||
1.0
|
||||
],
|
||||
"p_value": 1.0,
|
||||
"python_constraint_tool_use_rate": 1.0,
|
||||
"acceptance": {
|
||||
"code_accuracy_over_90_percent": true,
|
||||
"code_significantly_higher_than_pure": false,
|
||||
"all_code_trajectories_used_python_constraint": true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,552 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "5-2",
|
||||
"generated_at_utc": "2026-07-29T18:25:22.452238+00:00",
|
||||
"provider": "ark",
|
||||
"model": "doubao-seed-1-6-250615",
|
||||
"mode": "both",
|
||||
"tasks": 1,
|
||||
"dataset_manifest": {
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-2",
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"sampling": {
|
||||
"split": "test",
|
||||
"perturbations": [
|
||||
"perturbed_leaf",
|
||||
"perturbed_statement",
|
||||
"reorder_statement",
|
||||
"random_pair",
|
||||
"uncommon_name",
|
||||
"flip_role"
|
||||
],
|
||||
"people": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8
|
||||
],
|
||||
"per_cell": 2,
|
||||
"seed": 512,
|
||||
"cells": 42,
|
||||
"total": 84
|
||||
},
|
||||
"source_files": [
|
||||
{
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"published_rows": 76,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
11
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people3_num100.jsonl",
|
||||
"sha256": "4c3d767065edb62d19728da7a4e4e4c63f550c1719c3600c868f50f48715711e",
|
||||
"published_rows": 93,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people4_num100.jsonl",
|
||||
"sha256": "b3811d1ea4bbdb2ade55dde240361470516e0541c58b5c526609f99ad6a038d7",
|
||||
"published_rows": 94,
|
||||
"sampled_rows": [
|
||||
66,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people5_num100.jsonl",
|
||||
"sha256": "17529110798b0a659286983d0cf4e0d3b5a2b560dfb6ab3b3acb1b61ac9d7da7",
|
||||
"published_rows": 98,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
41
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people6_num100.jsonl",
|
||||
"sha256": "032c0c253bbe574335c741bb5ba7d07b48f622e217dc939f72a633ef9fbe46b0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
17,
|
||||
51
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people7_num100.jsonl",
|
||||
"sha256": "f62b1f0c3c45e3a62660951fc12b9716679f1466bdc313066d85d08d6fe3c767",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
79,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people8_num100.jsonl",
|
||||
"sha256": "01947375b3a16ce4fb340d6f0e1545d5aeadfc6f695c49cf2a6f436fcc5ba099",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
11,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people2_num100.jsonl",
|
||||
"sha256": "18479f9b2c327e9ce51e901479e9e7ffcd4397094e91fdca430871a3fc131fc3",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
94
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people3_num100.jsonl",
|
||||
"sha256": "cb8832b4aae19426e10a039236f0e4fb7dad797598edb3850db8067ff7cda1a6",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
24
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people4_num100.jsonl",
|
||||
"sha256": "ec8821279ff09db70098b97c5b47f452f2b9a6059b3d32ff22c9fce8342e4c30",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
27,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people5_num100.jsonl",
|
||||
"sha256": "791a14c3a7fea7713904786cdbd24ccc94901c0508c1ed3daf5abe3794857ed4",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
95,
|
||||
99
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people6_num100.jsonl",
|
||||
"sha256": "afadf008aa8aba0753bb198baca07ea51ebd9879045b619861225eea56559134",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
47,
|
||||
52
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people7_num100.jsonl",
|
||||
"sha256": "bd3900ae3527887659a1a0da1146780e0a47e4d96ff301e1c2039f35bcfff849",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
15,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people8_num100.jsonl",
|
||||
"sha256": "c3d91ba046bbeb5ec0d64c9147d3d2c76c1f631716da47d32b8379ebfcbe35a0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
86
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people2_num100.jsonl",
|
||||
"sha256": "c116f384f984150969554d783ccf7cecf4c6e56c59d2f5d326bfcfa722bb149c",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
60,
|
||||
76
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people3_num100.jsonl",
|
||||
"sha256": "ff772d57162fb500e608ef28c59eaf57ba1d80f60cf3f73902162c3476f6b0a2",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
64
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people4_num100.jsonl",
|
||||
"sha256": "be12e4b5a38792785264c16c11117f7f10f955517e32c92ea37949017a4da0cb",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people5_num100.jsonl",
|
||||
"sha256": "04a492522cdeb64a960632652b54e3fd27cc505fa8a1c48cf9c666cc8e7d6fb0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
55,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people6_num100.jsonl",
|
||||
"sha256": "4b6131d749c47ce1580d0443e373112cc5f0e20f1b0245455b3df63b106580c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
80,
|
||||
85
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people7_num100.jsonl",
|
||||
"sha256": "d3051b151ef7e5c47a9cd69b09b52f1ec94c7f899821ec685e286e578c7932c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
67,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people8_num100.jsonl",
|
||||
"sha256": "3710c66680d201ebf3d6d243fa9f54ee7e5ce35570ed44f87cd5111b05dc1d9b",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
2,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people2_num100.jsonl",
|
||||
"sha256": "92b9d18d12ea3eaaa0a49daa03f519b5c5f66614bf724e541e9c00e1b9be24ae",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
13,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people3_num100.jsonl",
|
||||
"sha256": "6eb60022244be6fa25d372d1580eb8dddfb06914d7290dbe1d64c9cb57c1f612",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
54,
|
||||
75
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people4_num100.jsonl",
|
||||
"sha256": "1cafedf8ea1c5bf393d2366a6d798e66730059fdd5290c8b2a8cca35de6c4589",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
90
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people5_num100.jsonl",
|
||||
"sha256": "9036535b0ace7b3cfe1b8cf94fbb898b0c07e8d88d23cbc914707ed6d0221ebc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
62,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people6_num100.jsonl",
|
||||
"sha256": "9d3a4bcd28f465b311bd5cdc96958b69529240f05e00a79d80ea7a54b8d53002",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
45
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people7_num100.jsonl",
|
||||
"sha256": "389713889ec22d14bf64d1bb9b846d1fb7a4adba44004b6ce9db1e0ac3d54190",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people8_num100.jsonl",
|
||||
"sha256": "8885fe0d63bfe5910617bd84e6be646f7b62a26978a649b4173889601d0f9229",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
71
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people2_num100.jsonl",
|
||||
"sha256": "25437adfc79438934dbaa113f4a2b06378d3f899a3cc9d8193202a3e0d746f29",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people3_num100.jsonl",
|
||||
"sha256": "087b47837520c8da31d78deac08781c3a62ad7961bae246d68f81843b3795ab0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
40
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people4_num100.jsonl",
|
||||
"sha256": "8e0edc42da3d5739a4e24f82725c6332e71189016391810b5e80d549c71d1004",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
44,
|
||||
57
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people5_num100.jsonl",
|
||||
"sha256": "c1e7ade7a1791ab17a1d951f7548ce8c54767190f0402e4e6785889810694f3f",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
5,
|
||||
47
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people6_num100.jsonl",
|
||||
"sha256": "4046b49fe876c7f4eab6eddc35ff0a9f01ad487faf59e9edb5605a39b3e56530",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
56
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people7_num100.jsonl",
|
||||
"sha256": "752a97113d999fcff5f2910c78fca761133822b99b5ae6ec9070421e0e51b2dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
34,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people8_num100.jsonl",
|
||||
"sha256": "951228b33ff7b4afffcbd247eeb67390d9af5ff2def9f9cdb0f8433564fffc15",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
12,
|
||||
68
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people2_num100.jsonl",
|
||||
"sha256": "2885eb7b356f9ab67ff2ccef9372dcfa7ebcf6ed2b8c5e720db1d74f03840c8d",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
7,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people3_num100.jsonl",
|
||||
"sha256": "31026489710814617c66a6879408accc099c34f395425b41d0bd56743d933461",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people4_num100.jsonl",
|
||||
"sha256": "a5d56cc8724a40fd674dba7730f52ca8dec26966f07ff434302bd62b1260a458",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people5_num100.jsonl",
|
||||
"sha256": "1429f88b80794e12f731bcd9cd2502384a3c4ac1a3292010103baccd86e93ae7",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
73
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people6_num100.jsonl",
|
||||
"sha256": "ccfee7ea7e2f24376e88193a979a20ac6410ec2458c534af7e4daedab4ff47ff",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
39,
|
||||
62
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people7_num100.jsonl",
|
||||
"sha256": "b5720e5a434747f323f90086ac6bae7ba33d1b6ba8d617424f599bfabdbcdeca",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
1,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people8_num100.jsonl",
|
||||
"sha256": "d668973f128b119ad065e0b58f33099516dbd801f0e7957c00e876115e8119dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
96
|
||||
]
|
||||
}
|
||||
],
|
||||
"label_validation": "all rows independently solved with python-constraint"
|
||||
},
|
||||
"dataset_manifest_sha256": "2e55a7d8b07e55b035a8129889b11dff186589bcad97c02c0ce553f38249212f",
|
||||
"pure": [
|
||||
{
|
||||
"id": "perturbed_leaf-p2-r006",
|
||||
"num": 2,
|
||||
"pred": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"gold": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"correct": true,
|
||||
"source": {
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"config": "test",
|
||||
"split": "perturbed_leaf",
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"file_sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"row": 6,
|
||||
"dataset_index": 9
|
||||
},
|
||||
"codes": [],
|
||||
"text": "Let's analyze the statements of James and Sophia step by step:\n\n\n### **Step 1: Analyze Sophia's Statement** \nSophia claims: \"James is a knight if and only if James is a knave.\" \nAn \"if and only if\" (↔) statement is true only when both sides have the same truth value. Here, the two sides are \"James is a knight\" (J_knight) and \"James is a knave\" (¬J_knight, since a knave is not a knight). \n\n- If J_knight is **true**, then ¬J_knight is **false**. Thus, \"true ↔ false\" is **false**. \n- If J_knight is **false**, then ¬J_knight is **true**. Thus, \"false ↔ true\" is **false**. \n\nIn all cases, Sophia’s statement is **logically false** (a contradiction). \n\n\n### **Step 2: Determine Sophia's Identity** \nSince Sophia’s statement is false, and knaves always lie, Sophia must be a **knave**. \n\n\n### **Step 3: Analyze James's Statement** \nJames claims: \"Sophia is not a knight.\" \nWe already established Sophia is a knave, so \"Sophia is not a knight\" is a **true statement**. \n\nSince James’s statement is true, and knights always tell the truth, James must be a **knight**. \n\n\n### **Verification** \n- James (knight) tells the truth: \"Sophia is not a knight\" is true (Sophia is a knave). \n- Sophia (knave) lies: Her statement \"James is a knight if and only if James is a knave\" is false (consistent with knaves lying). \n\n\n{\"James\": \"knight\", \"Sophia\": \"knave\"}",
|
||||
"used_python_constraint": false,
|
||||
"duration_s": 33.894,
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "021785349436115840fd37cfe82fdd108ca4a9d2a6525759244e1",
|
||||
"response_model": "doubao-seed-1-6-250615",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 265,
|
||||
"completion_tokens": 1351,
|
||||
"total_tokens": 1616,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"pure_acc": 1.0,
|
||||
"code": [
|
||||
{
|
||||
"id": "perturbed_leaf-p2-r006",
|
||||
"num": 2,
|
||||
"pred": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"gold": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"correct": true,
|
||||
"source": {
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"config": "test",
|
||||
"split": "perturbed_leaf",
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"file_sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"row": 6,
|
||||
"dataset_index": 9
|
||||
},
|
||||
"codes": [
|
||||
"from constraint import Problem\n\np = Problem()\n# 为James和Sophia创建布尔变量，True=骑士，False=无赖\np.addVariable('James', [True, False])\np.addVariable('Sophia', [True, False])\n\n# James说：\"Sophia不是骑士\" → James的身份等价于(Sophia是无赖)\n# 即 James == (not Sophia)\np.addConstraint(lambda j, s: j == (not s), ['James', 'Sophia'])\n\n# Sophia说：\"James是骑士当且仅当James是无赖\" → Sophia的身份等价于(James是骑士 ⇨⇦ James是无赖)\n# 即 Sophia == (James == (not James))\np.addConstraint(lambda j, s: s == (j == (not j)), ['James', 'Sophia'])\n\n# 求解并输出结果\nfor solution in p.getSolutions():\n result = {k: 'knight' if v else 'knave' for k, v in solution.items()}\n print(result)"
|
||||
],
|
||||
"text": "{\"James\": \"knight\", \"Sophia\": \"knave\"}",
|
||||
"used_python_constraint": true,
|
||||
"duration_s": 52.753,
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "021785349469880840fd37cfe82fdd108ca4a9d2a652575539cd2",
|
||||
"response_model": "doubao-seed-1-6-250615",
|
||||
"finish_reason": "tool_calls",
|
||||
"usage": {
|
||||
"prompt_tokens": 1152,
|
||||
"completion_tokens": 1827,
|
||||
"total_tokens": 2979,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 1
|
||||
},
|
||||
{
|
||||
"turn": 2,
|
||||
"response_id": "021785349507332840fd37cfe82fdd108ca4a9d2a652575ed09ce",
|
||||
"response_model": "doubao-seed-1-6-250615",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 1451,
|
||||
"completion_tokens": 515,
|
||||
"total_tokens": 1966,
|
||||
"cached_prompt_tokens": 0
|
||||
},
|
||||
"tool_calls": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"code_acc": 1.0,
|
||||
"paired_analysis": {
|
||||
"test": "two-sided exact McNemar/binomial test on discordant pairs",
|
||||
"n": 1,
|
||||
"contingency": {
|
||||
"pure_only": 0,
|
||||
"code_only": 0,
|
||||
"discordant": 0
|
||||
},
|
||||
"pure_accuracy": 1.0,
|
||||
"code_accuracy": 1.0,
|
||||
"accuracy_delta": 0.0,
|
||||
"code_accuracy_wilson_95": [
|
||||
0.20654931437723745,
|
||||
1.0
|
||||
],
|
||||
"p_value": 1.0,
|
||||
"python_constraint_tool_use_rate": 1.0,
|
||||
"acceptance": {
|
||||
"code_accuracy_over_90_percent": true,
|
||||
"code_significantly_higher_than_pure": false,
|
||||
"all_code_trajectories_used_python_constraint": true
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,531 @@
|
||||
{
|
||||
"schema_version": "2.0",
|
||||
"experiment": "5-2",
|
||||
"generated_at_utc": "2026-07-29T18:34:11.952867+00:00",
|
||||
"provider": "ollama",
|
||||
"model": "qwen3:1.7b",
|
||||
"mode": "both",
|
||||
"tasks": 1,
|
||||
"dataset_manifest": {
|
||||
"schema_version": "1.0",
|
||||
"experiment": "5-2",
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"sampling": {
|
||||
"split": "test",
|
||||
"perturbations": [
|
||||
"perturbed_leaf",
|
||||
"perturbed_statement",
|
||||
"reorder_statement",
|
||||
"random_pair",
|
||||
"uncommon_name",
|
||||
"flip_role"
|
||||
],
|
||||
"people": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
5,
|
||||
6,
|
||||
7,
|
||||
8
|
||||
],
|
||||
"per_cell": 2,
|
||||
"seed": 512,
|
||||
"cells": 42,
|
||||
"total": 84
|
||||
},
|
||||
"source_files": [
|
||||
{
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"published_rows": 76,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
11
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people3_num100.jsonl",
|
||||
"sha256": "4c3d767065edb62d19728da7a4e4e4c63f550c1719c3600c868f50f48715711e",
|
||||
"published_rows": 93,
|
||||
"sampled_rows": [
|
||||
6,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people4_num100.jsonl",
|
||||
"sha256": "b3811d1ea4bbdb2ade55dde240361470516e0541c58b5c526609f99ad6a038d7",
|
||||
"published_rows": 94,
|
||||
"sampled_rows": [
|
||||
66,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people5_num100.jsonl",
|
||||
"sha256": "17529110798b0a659286983d0cf4e0d3b5a2b560dfb6ab3b3acb1b61ac9d7da7",
|
||||
"published_rows": 98,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
41
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people6_num100.jsonl",
|
||||
"sha256": "032c0c253bbe574335c741bb5ba7d07b48f622e217dc939f72a633ef9fbe46b0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
17,
|
||||
51
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people7_num100.jsonl",
|
||||
"sha256": "f62b1f0c3c45e3a62660951fc12b9716679f1466bdc313066d85d08d6fe3c767",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
79,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_leaf/people8_num100.jsonl",
|
||||
"sha256": "01947375b3a16ce4fb340d6f0e1545d5aeadfc6f695c49cf2a6f436fcc5ba099",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
11,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people2_num100.jsonl",
|
||||
"sha256": "18479f9b2c327e9ce51e901479e9e7ffcd4397094e91fdca430871a3fc131fc3",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
94
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people3_num100.jsonl",
|
||||
"sha256": "cb8832b4aae19426e10a039236f0e4fb7dad797598edb3850db8067ff7cda1a6",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
24
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people4_num100.jsonl",
|
||||
"sha256": "ec8821279ff09db70098b97c5b47f452f2b9a6059b3d32ff22c9fce8342e4c30",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
27,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people5_num100.jsonl",
|
||||
"sha256": "791a14c3a7fea7713904786cdbd24ccc94901c0508c1ed3daf5abe3794857ed4",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
95,
|
||||
99
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people6_num100.jsonl",
|
||||
"sha256": "afadf008aa8aba0753bb198baca07ea51ebd9879045b619861225eea56559134",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
47,
|
||||
52
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people7_num100.jsonl",
|
||||
"sha256": "bd3900ae3527887659a1a0da1146780e0a47e4d96ff301e1c2039f35bcfff849",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
15,
|
||||
28
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/perturbed_statement/people8_num100.jsonl",
|
||||
"sha256": "c3d91ba046bbeb5ec0d64c9147d3d2c76c1f631716da47d32b8379ebfcbe35a0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
86
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people2_num100.jsonl",
|
||||
"sha256": "c116f384f984150969554d783ccf7cecf4c6e56c59d2f5d326bfcfa722bb149c",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
60,
|
||||
76
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people3_num100.jsonl",
|
||||
"sha256": "ff772d57162fb500e608ef28c59eaf57ba1d80f60cf3f73902162c3476f6b0a2",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
0,
|
||||
64
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people4_num100.jsonl",
|
||||
"sha256": "be12e4b5a38792785264c16c11117f7f10f955517e32c92ea37949017a4da0cb",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people5_num100.jsonl",
|
||||
"sha256": "04a492522cdeb64a960632652b54e3fd27cc505fa8a1c48cf9c666cc8e7d6fb0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
55,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people6_num100.jsonl",
|
||||
"sha256": "4b6131d749c47ce1580d0443e373112cc5f0e20f1b0245455b3df63b106580c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
80,
|
||||
85
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people7_num100.jsonl",
|
||||
"sha256": "d3051b151ef7e5c47a9cd69b09b52f1ec94c7f899821ec685e286e578c7932c5",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
67,
|
||||
92
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/reorder_statement/people8_num100.jsonl",
|
||||
"sha256": "3710c66680d201ebf3d6d243fa9f54ee7e5ce35570ed44f87cd5111b05dc1d9b",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
2,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people2_num100.jsonl",
|
||||
"sha256": "92b9d18d12ea3eaaa0a49daa03f519b5c5f66614bf724e541e9c00e1b9be24ae",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
13,
|
||||
14
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people3_num100.jsonl",
|
||||
"sha256": "6eb60022244be6fa25d372d1580eb8dddfb06914d7290dbe1d64c9cb57c1f612",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
54,
|
||||
75
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people4_num100.jsonl",
|
||||
"sha256": "1cafedf8ea1c5bf393d2366a6d798e66730059fdd5290c8b2a8cca35de6c4589",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
90
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people5_num100.jsonl",
|
||||
"sha256": "9036535b0ace7b3cfe1b8cf94fbb898b0c07e8d88d23cbc914707ed6d0221ebc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
62,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people6_num100.jsonl",
|
||||
"sha256": "9d3a4bcd28f465b311bd5cdc96958b69529240f05e00a79d80ea7a54b8d53002",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
45
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people7_num100.jsonl",
|
||||
"sha256": "389713889ec22d14bf64d1bb9b846d1fb7a4adba44004b6ce9db1e0ac3d54190",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/random_pair/people8_num100.jsonl",
|
||||
"sha256": "8885fe0d63bfe5910617bd84e6be646f7b62a26978a649b4173889601d0f9229",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
68,
|
||||
71
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people2_num100.jsonl",
|
||||
"sha256": "25437adfc79438934dbaa113f4a2b06378d3f899a3cc9d8193202a3e0d746f29",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
16,
|
||||
82
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people3_num100.jsonl",
|
||||
"sha256": "087b47837520c8da31d78deac08781c3a62ad7961bae246d68f81843b3795ab0",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
37,
|
||||
40
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people4_num100.jsonl",
|
||||
"sha256": "8e0edc42da3d5739a4e24f82725c6332e71189016391810b5e80d549c71d1004",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
44,
|
||||
57
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people5_num100.jsonl",
|
||||
"sha256": "c1e7ade7a1791ab17a1d951f7548ce8c54767190f0402e4e6785889810694f3f",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
5,
|
||||
47
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people6_num100.jsonl",
|
||||
"sha256": "4046b49fe876c7f4eab6eddc35ff0a9f01ad487faf59e9edb5605a39b3e56530",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
8,
|
||||
56
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people7_num100.jsonl",
|
||||
"sha256": "752a97113d999fcff5f2910c78fca761133822b99b5ae6ec9070421e0e51b2dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
34,
|
||||
69
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/uncommon_name/people8_num100.jsonl",
|
||||
"sha256": "951228b33ff7b4afffcbd247eeb67390d9af5ff2def9f9cdb0f8433564fffc15",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
12,
|
||||
68
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people2_num100.jsonl",
|
||||
"sha256": "2885eb7b356f9ab67ff2ccef9372dcfa7ebcf6ed2b8c5e720db1d74f03840c8d",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
7,
|
||||
80
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people3_num100.jsonl",
|
||||
"sha256": "31026489710814617c66a6879408accc099c34f395425b41d0bd56743d933461",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
79
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people4_num100.jsonl",
|
||||
"sha256": "a5d56cc8724a40fd674dba7730f52ca8dec26966f07ff434302bd62b1260a458",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
36,
|
||||
97
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people5_num100.jsonl",
|
||||
"sha256": "1429f88b80794e12f731bcd9cd2502384a3c4ac1a3292010103baccd86e93ae7",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
22,
|
||||
73
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people6_num100.jsonl",
|
||||
"sha256": "ccfee7ea7e2f24376e88193a979a20ac6410ec2458c534af7e4daedab4ff47ff",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
39,
|
||||
62
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people7_num100.jsonl",
|
||||
"sha256": "b5720e5a434747f323f90086ac6bae7ba33d1b6ba8d617424f599bfabdbcdeca",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
1,
|
||||
91
|
||||
]
|
||||
},
|
||||
{
|
||||
"path": "test/flip_role/people8_num100.jsonl",
|
||||
"sha256": "d668973f128b119ad065e0b58f33099516dbd801f0e7957c00e876115e8119dc",
|
||||
"published_rows": 100,
|
||||
"sampled_rows": [
|
||||
64,
|
||||
96
|
||||
]
|
||||
}
|
||||
],
|
||||
"label_validation": "all rows independently solved with python-constraint"
|
||||
},
|
||||
"dataset_manifest_sha256": "2e55a7d8b07e55b035a8129889b11dff186589bcad97c02c0ce553f38249212f",
|
||||
"pure": [
|
||||
{
|
||||
"id": "perturbed_leaf-p2-r006",
|
||||
"num": 2,
|
||||
"pred": null,
|
||||
"gold": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"correct": false,
|
||||
"source": {
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"config": "test",
|
||||
"split": "perturbed_leaf",
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"file_sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"row": 6,
|
||||
"dataset_index": 9
|
||||
},
|
||||
"codes": [],
|
||||
"text": "{\"A\": \"knight\", \"B\": \"knave\"}",
|
||||
"used_python_constraint": false,
|
||||
"duration_s": 7.214,
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "chatcmpl-321",
|
||||
"response_model": "qwen3:1.7b",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 191,
|
||||
"completion_tokens": 702,
|
||||
"total_tokens": 893,
|
||||
"cached_prompt_tokens": null
|
||||
},
|
||||
"tool_calls": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"pure_acc": 0.0,
|
||||
"code": [
|
||||
{
|
||||
"id": "perturbed_leaf-p2-r006",
|
||||
"num": 2,
|
||||
"pred": null,
|
||||
"gold": {
|
||||
"James": "knight",
|
||||
"Sophia": "knave"
|
||||
},
|
||||
"correct": false,
|
||||
"source": {
|
||||
"dataset": "K-and-K/perturbed-knights-and-knaves",
|
||||
"revision": "bc7ee75a15ee8196ccbdb7df3ab46284340412e2",
|
||||
"license": "CC-BY-NC-SA-4.0",
|
||||
"config": "test",
|
||||
"split": "perturbed_leaf",
|
||||
"path": "test/perturbed_leaf/people2_num100.jsonl",
|
||||
"file_sha256": "541c5dadb60c54e4c9973cafa0dbd69635538fb44aa5dcb28dc64ea7a78a56be",
|
||||
"row": 6,
|
||||
"dataset_index": 9
|
||||
},
|
||||
"codes": [],
|
||||
"text": "{\"A\": \"knight\", \"B\": \"knave\"}\n\n```python\nfrom constraint import Problem\np = Problem()\n\n# Add variables for James and Sophia\np.addVariable(\"James\", [True, False])\np.addVariable(\"Sophia\", [True, False])\n\n# James says \"Sophia is not a knight\"\np.addConstraint(lambda j, s: j == (not s), [\"James\", \"Sophia\"])\n\n# Sophia says \"James is a knight if and only if James is a knave\"\np.addConstraint(lambda j: j == (j == not j), [\"James\"])\n\nfor solution in p.getSolutions():\n print({k: 'knight' if v else 'knave' for k, v in solution.items()})\n```\n\nThe solution shows James is a knight (truth-teller) and Sophia is a knave (liar). The constraints are satisfied:\n- James's statement \"Sophia is not a knight\" is true (she's a knave).\n- Sophia's statement \"James is a knight iff James is a knave\" is false (since James is a knight, the statement is \"True iff False\", which is false, consistent with her being a knave.\n\nThe code returns:\n{\"A\": \"knight\", \"B\": \"knave\"}",
|
||||
"used_python_constraint": false,
|
||||
"duration_s": 7.145,
|
||||
"provider_receipts": [
|
||||
{
|
||||
"turn": 1,
|
||||
"response_id": "chatcmpl-732",
|
||||
"response_model": "qwen3:1.7b",
|
||||
"finish_reason": "stop",
|
||||
"usage": {
|
||||
"prompt_tokens": 831,
|
||||
"completion_tokens": 863,
|
||||
"total_tokens": 1694,
|
||||
"cached_prompt_tokens": null
|
||||
},
|
||||
"tool_calls": 0
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"code_acc": 0.0,
|
||||
"paired_analysis": {
|
||||
"test": "two-sided exact McNemar/binomial test on discordant pairs",
|
||||
"n": 1,
|
||||
"contingency": {
|
||||
"pure_only": 0,
|
||||
"code_only": 0,
|
||||
"discordant": 0
|
||||
},
|
||||
"pure_accuracy": 0.0,
|
||||
"code_accuracy": 0.0,
|
||||
"accuracy_delta": 0.0,
|
||||
"code_accuracy_wilson_95": [
|
||||
0.0,
|
||||
0.7934506856227626
|
||||
],
|
||||
"p_value": 1.0,
|
||||
"python_constraint_tool_use_rate": 0.0,
|
||||
"acceptance": {
|
||||
"code_accuracy_over_90_percent": false,
|
||||
"code_significantly_higher_than_pure": false,
|
||||
"all_code_trajectories_used_python_constraint": false
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user