liqiang/ai-agent-book
1
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
Details
dependency resolution / resolve (3.11) (push) Canceled after 0s
Details
dependency resolution / resolve (3.13) (push) Canceled after 0s
Details
deploy-pages / build (push) Canceled after 0s
Details
deploy-pages / deploy (push) Canceled after 0s
Details
i18n consistency check / check (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
Details
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
Details
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
Details
web-search-agent tests / test (push) Canceled after 0s
Details
web-search-agent tests / agentbook (push) Canceled after 0s
Details

Browse Source
This commit is contained in:
liqiang
2026-08-20 13:12:50 +00:00
commit b119135836
10275 changed files with 3284984 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
chapter8/AWorld-train/README.md
+2439
View File
File diff suppressed because it is too large Load Diff
chapter8/AdaptThink/README.md
+1120
View File
File diff suppressed because it is too large Load Diff
chapter8/AdaptThink/TRAINING_REPORT.md
+119
View File
@@ -0,0 +1,119 @@
# Experiment 8-10: AdaptThink training report
This report records the historical AdaptThink 1.5B, δ=0.05 training run used by
the book. It is a training report, not a fresh local reproduction. In accordance
with the book's distribution policy, model checkpoints are not distributed.
## Public runs
- Main training run: [`wubbn5tj`](https://wandb.ai/bojieli-pine-ai/adapt_think_verl/runs/wubbn5tj)
- Baseline-only run: [`dblyx7cm`](https://wandb.ai/bojieli-pine-ai/adapt_think_verl/runs/dblyx7cm)
- W&B project: [`bojieli-pine-ai/adapt_think_verl`](https://wandb.ai/bojieli-pine-ai/adapt_think_verl)
The main run contains 411 training-history rows for steps 0–410 and 42
validation rows at step 0 and every 10 steps through step 410. The baseline run
contains the same step-0 validation metrics as the main run.
## Training configuration
| Item | Recorded value |
| --- | --- |
| Base model | `deepseek-ai/DeepSeek-R1-Distill-Qwen-1.5B` |
| Historical source commit | `9e588202ff56fe93cdbe49f5594cf895f7d6b7c2` |
| Hardware | 8 × NVIDIA H100 80GB HBM3 |
| Runtime environment | CUDA 12.6, Python 3.13.7 |
| Training data | DeepScaler |
| Batch size | 128 |
| Rollouts per prompt | 16 |
| Prompt / response limit | 1,024 / 16,384 tokens |
| NoThinking response limit | 4,096 tokens |
| Learning rate | `2e-6` |
| NoThinking bonus δ | 0.05 |
| Save / validation interval | Every 10 steps |
| Configured schedule | 10 epochs, 3,140 optimizer steps |
| Selected report point | Step 300, approximately 28.37 hours |
| Last retained point | Step 410, approximately 36.92 hours |
| Final W&B state | `crashed` |
The run therefore did not finish its configured ten-epoch schedule. The crash
occurred after the selected step-300 report point.
## Step-300 result
The book uses step 300 as the comparison point. Accuracy and response length are
the aggregate validation metrics logged by the main W&B run.
| Dataset | Accuracy, step 0 | Accuracy, step 300 | Change | Mean response length, step 0 | Mean response length, step 300 | Reduction | NoThinking at step 300 |
| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| GSM8K | 0.79681577 | 0.81880212 | +2.1986 pp | 1,025.2350 | 477.3275 | 53.44% | 84.15% |
| MATH500 | 0.81000000 | 0.81800000 | +0.8000 pp | 4,911.4600 | 1,576.6220 | 67.90% | 83.80% |
| AIME2024 mean@16 | 0.31458333 | 0.31041667 | -0.4167 pp | 12,119.5063 | 6,402.2271 | 47.17% | 56.25% |
Mean response length fell substantially on all three datasets. Accuracy improved
on GSM8K and MATH500 but declined slightly on AIME2024, so this run does not
support a claim of uniform accuracy improvement.
### Conditional step-300 aggregates
| Dataset | NoThinking accuracy | Thinking accuracy | NoThinking response length | Thinking response length |
| --- | ---: | ---: | ---: | ---: |
| GSM8K | 0.81621622 | 0.83253589 | 359.6414 | 1,102.3589 |
| MATH500 | 0.82338902 | 0.79012346 | 1,089.7446 | 4,095.1605 |
| AIME2024 mean@16 | 0.28680561 | 0.40963620 | 4,392.5965 | 8,927.4171 |
The lower NoThinking rate on AIME2024 is consistent with difficulty-sensitive
routing at the dataset level. Aggregate metrics do not prove that the model chose
the correct mode for every individual problem.
## Later retained telemetry
Step 410 is shown separately because it is not the book's selected checkpoint.
| Dataset | Accuracy at step 410 | Mean response length | NoThinking ratio |
| --- | ---: | ---: | ---: |
| GSM8K | 0.818044 | 464.56 | 82.03% |
| MATH500 | 0.852000 | 1,481.91 | 74.80% |
| AIME2024 mean@16 | 0.318750 | 5,873.74 | 49.79% |
## Evaluation protocol represented by the logs
- Maximum response length: 16,384 tokens.
- Sampling temperature: 0.6; top-p: 0.95.
- GSM8K and MATH500 use one sampled response per problem.
- AIME2024 uses 16 sampled responses per problem and reports mean@16.
- Answers are graded using the project's boxed-answer rule-based grader.
These are in-training validation metrics. They are not results from a separately
retained post-conversion evaluation run.
## Checkpoint and provenance boundary
The step-300 history includes a checkpoint-save timing event, but the checkpoint
is not distributed with the book. There is also no public receipt showing that
this historical checkpoint was converted and evaluated by `run_eval_verl_hf.sh`,
and no retained MMLU rerun.
The W&B main run records source commit
`9e588202ff56fe93cdbe49f5594cf895f7d6b7c2`. The repository's future
reproduction instructions pin its direct child
`0033ad172dd53ac64004b763477407014f21b838`; the preprocessing, training, and
evaluation entrypoints are unchanged between those commits.
One manual correction is required for a future train-to-evaluate run. The
training script interpolates an undefined `adapt_think_max_response_length` into
the experiment name, producing a `-fl-` path segment. The evaluation script
instead expects `-fl4096` and a different checkpoint directory layout.
## Limitations
- This is one historical run, not a multi-seed replication.
- No per-example step-300 predictions, RNG state, or complete main-run stdout was
retained.
- GSM8K and MATH500 use stochastic single-sample validation.
- No confidence intervals or statistical-significance claims are provided.
- Checkpoint selection and reporting use the same validation suites.
- The results support a descriptive account of the logged run, not a causal or
universal claim about difficulty awareness.
Within those boundaries, Experiment 8-10 is complete as a checkpoint-free
training report.
chapter8/EXPERIMENT_LEDGER.md
+33
View File
@@ -0,0 +1,33 @@
# Chapter 8 experiment coverage ledger
Training-paper reproduction guides are tracked separately from completed local
runs. Paper numbers, copied logs, static scripts, and source checkouts do not
prove that a checkpoint was trained or evaluated in this workspace.
| Experiment | Manuscript acceptance scope | Current evidence | Audit status |
| --- | --- | --- | --- |
| 8-1 | 10,000-episode Q-learning curve and 100-episode greedy evaluation in the treasure environment | Cross-chapter [`learning-from-experience`](../chapter1/learning-from-experience/) evidence contains the deterministic Q-learning arm and completion gates. | **Complete** |
| 8-2 | Same environment comparison with Kimi K3, including the first successful trajectory and no fallback | [`validation/20260730_011704/evidence.json`](../chapter1/learning-from-experience/validation/20260730_011704/evidence.json) retains 17/17 raw API receipts and the first-game trajectory. | **Complete** |
| 8-3 | Train 100M MiniMind through pretrain, SFT, and preference optimization; compare QK Norm + Muon | [`exp8-3-training-report-20260731-v1`](MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/manifest.json) hashes all 49 historical outputs across the six arm/stage cells and eight preregistered arm-blind ARK judgments with raw requests/responses, unique IDs, usage, and latency. The independent audit scored QK-Norm + Muon 3.6250 versus original 2.0417 overall (+1.5833; 7 wins, 1 tie). The report freezes the exact MiniMind source revision and relevant source-file hashes, a dataset revision with all three Git-LFS hashes/sizes, the book environment lock, and six future reproduction commands. Historical source/data/checkpoint identities and stepwise loss logs were not retained, so the 36-vs-12-step and 2.0-vs-1.7 loss observations remain explicitly qualified historical claims. Checkpoints are intentionally local and are not acceptance artifacts. | **Complete evidence-backed training report** |
| 8-4 | Train VLM projection alignment then SFT and evaluate vision-language outputs | [`exp8-4-training-report-20260731-v1`](MiniMind-pretrain/validation/runs/exp8-4-training-report-20260731-v1/manifest.json) hashes all 64 historical outputs from eight configurations × eight images, embeds the exact eight hash-pinned evaluation images in eight anonymous image-aware ARK judge requests, and retains raw responses, unique IDs, usage, latency, source/data/CLIP pins, future commands, and provenance limits. The judge ranked original/SFT highest at 1.9062; the matched SFT-base QK-Norm+Muon comparisons were lower by 0.1876 after projection training and 0.6250 after full VLM SFT, so the book's optimizer-advantage claim is not forced. Historical revisions/checkpoints were not retained; checkpoints intentionally remain local and are not acceptance artifacts. | **Complete evidence-backed training report** |
| 8-5 | Korean continued pretraining plus Korean instruction SFT, with Korean gain and English holdout comparison | [`exp8-5-training-report-20260731-v1`](continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/manifest.json) hashes the historical RTX-4090 report, current training/evaluation sources, all 15 retained outputs, five stage-blind ARK judgments with raw response IDs/usage/latency, and an immutable future reproduction contract. Final-minus-baseline Korean mean was +1.7777; English fell 0.8333 within the declared 1.0 tolerance; the materially false kimchi answer is explicit. Historical upstream revisions/seeds were not retained and current pins are not misrepresented as historical. Checkpoints are intentionally local and are not acceptance artifacts. | **Complete evidence-backed training report** |
| 8-6 | Train/evaluate Orpheus cross-sentence voice consistency and Sesame paralinguistic tags, including failure comparisons | Completed local RTX PRO 6000 campaign: both LoRAs received 60 optimizer updates on substantive real-speech subsets with held-out loss evaluation; 40 matched base/adapted WAVs, adapter hashes/identities, AudioSet/MFCC proxy comparisons, and negative cases are retained in [`speech-sft-experiment/validation/exp8-6-20260804-v1/`](speech-sft-experiment/validation/exp8-6-20260804-v1/). Full adapters: [Orpheus](https://huggingface.co/bojieli/exp7-6-orpheus-elise-lora) and [Sesame](https://huggingface.co/bojieli/exp7-6-sesame-elise-tags-lora). The report separates execution completion from quality hypotheses and makes no perceptual-quality claim. | **Complete—bounded GPU campaign** |
| 8-7 | SFT gpt-oss-20b for selectable reasoning language and test zero-shot Chinese plus trained languages | `MultilingualReasoning/gpt_oss_20b_sft.py` implements training. No checkpoint or before/after multilingual benchmark. | **Incomplete—GPU training** |
| 8-8 | Generate teacher outputs, train prompt-distilled student, and compare teacher/student quality, latency, and cost | The campaign in `chapter8/prompt-distillation/validation/exp8-8-kimi3-smollm2-20260730/` retains 160/160 training and 80/80 held-out real Kimi K3 receipts, a real CUDA-trained SmolLM2-135M-Instruct LoRA checkpoint, the training receipt, and the paired comparison. Held-out: teacher 100%, baseline 0%, trained 95%; ~197× latency speedup; ~75% input-token reduction; eight of eight evidence gates pass. | **Complete saved campaign** |
| 8-9 | Rejection-sample verified teacher CoT, SFT a student, compare baseline/student/teacher, and inspect reflection/backtracking/verification | All 24 real Kimi K3 AIME cases retain completed trajectories. The deterministic verifier accepted 23 for SFT and rejected `aime-2016-9-I`, whose native low-reasoning retry completed with the wrong answer. Real CUDA SFT produced checkpoint `exp8-9-qwen25-1.5b-kimi-k3-20260801-v1`; `experiment_8_9_complete_20260803_v2.json` retains the full three-arm comparison: baseline 1/24, student 2/24, teacher 23/24, paired p=1.0, about 4.5% teacher-capability recovery, and inspected reflection/backtracking/verification rates. | **Complete saved campaign; uplift not significant** |
| 8-10 | AdaptThink training and evaluation of Thinking/NoThinking routing | Checkpoint-free [training report](AdaptThink/TRAINING_REPORT.md) records public W&B runs `wubbn5tj` (main) and `dblyx7cm` (step-0 baseline), 411 history rows through step 410, the 8×H100/CUDA 12.6 environment, source revisions, and exact step-0→300 metrics. At step 300, response length fell 67.90%/53.44%/47.17% on MATH500/GSM8K/AIME; accuracy changed +0.80/+2.20/-0.42 pp, so no uniform gain is claimed. The run continued past the selected point and crashed at 410. Checkpoints, per-example outputs, and an independent successful checkpoint/MMLU evaluation receipt were not retained; the advertised evaluation path also requires manual correction. | **Complete checkpoint-free training report** |
| 8-11 | GeneralPoints language/VL SFT-vs-PPO ID/OOD comparison under equal budget | Authoritative `bojieli/SFTvsRL` checkout matches `fef0a4a…`; exact GP train/eval scripts mapped, no checkpoint/run. | **External reproduction; not run** |
| 8-12 | V-IRL-VL PPO navigation with ID/rule-OOD/visual-OOD evaluation | Same pinned SFTvsRL checkout is the real source; `SpatialReasoning/` is a guide, not a separate implementation. No training/evaluation run. | **External reproduction; not run** |
| 8-13 | SimpleVLA-RL LIBERO/RoboTwin training/evaluation, including result reward and emergent policy evidence | Pinned `PRIME-RL/SimpleVLA-RL` checkout at `7c51662…`. Checkpoint placeholders, simulator/assets, and full CUDA lock remain unresolved; no run. | **External reproduction; dependency contract incomplete** |
| 8-14 | ReTool SFT warmup + PPO with live SandboxFusion execution and AIME comparison | veRL checkout matches `1593fc3…`; README pins SandboxFusion `4a0d573…`, which is absent locally. No sandbox service, SFT checkpoint, PPO run, or evaluation. | **External reproduction; not run** |
| 8-15 | Run AWorld MCP reset/episode loop and train Qwen3-4B until reward/tool-use improves | AWorld and veRL checkouts match the pinned SHAs and exact entrypoints are mapped. Historical upstream logs in the checkout do not establish a current run; no local reward curve/checkpoint. | **External reproduction; not run** |
| 8-16 | RLVP GRPO baseline vs verified path signals on TerminalBench and miniF2F over required seeds | README pins `19PINE-AI/rlvp` at `1ad30bc…` and exact train/eval sequence; checkout and CUDA results are absent. | **External reproduction; not run** |
| 8-17 | Build premature-completion DPO preference pairs from coding-agent bad cases, run 7B LoRA DPO training, and compare the unfinished-task set with the completed-task holdout | [`premature-completion-dpo/`](premature-completion-dpo/) contains 24 bad cases, a held-out evaluation set separated into 12 unfinished tasks and 8 completed-task holdouts, deterministic and teacher-model preference-pair builders, TRL DPO/GRPO training scripts, and a deterministic evaluator with a mock mode. On the local RTX PRO 6000, fixed-candidate decision comparison improved unfinished-task correctness from 3/12 (25.0%) to 11/12 (91.7%) while completed-task holdout correctness stayed 8/8 (100%). The report records the free-generation supplement and the rejected control-pair iteration. | **Complete—GPU run and evidence saved** |
| 8-18 | Convert the Chinese curved-quote bad case into a scope-sensitive Skill, synthesize protected-region examples, run open-weight LoRA SFT, and validate Chinese prose conversion without mutating English/code/JSON | [`curly-quote-sft/`](curly-quote-sft/) contains deterministic data blueprints, 16 scope kinds, 10 article types, 9 code languages, a manual audit record, and [manifest](curly-quote-sft/validation/manifest.json). RTX PRO 6000 run: 1024 train / 256 held-out / 256 boundary examples, 2 epochs and 256 updates; after explicit positive/negative scope rules, adapted exact is 96.9% / 97.7%, with 100% protected-region preservation. JSON remains 68.8%, so the campaign still requires a dedicated structured-data track. | **Complete—GPU run and evidence saved** |
| 8-19 | Diagnose `old_string`/special-string failures, train exact-copy behavior on unseen random strings and tool JSON arguments, and measure byte-exact holdout/boundary retention | [`exact-copy-sft/`](exact-copy-sft/) contains deterministic random-string generation, 10 language contexts, decoy/tool-call tasks, Qwen3-8B bf16 LoRA training, and [manifest](exact-copy-sft/validation/manifest.json). RTX PRO 6000 run: 1024 train / 256 held-out / 256 boundary examples, 2 epochs; byte-exact held-out accuracy 37.5% → 78.9%, boundary 80.1% (mean first-diff byte 54.0 / 54.2). A separate 512-probe audit covers Qwen3, Qwen2.5 and Mistral tokenizers. | **Complete—GPU run and evidence saved** |
Pinned source identities and acquisition commands are maintained in
[README.md](README.md). The current host has executed the NVIDIA/CUDA
experiments on an RTX PRO 6000 Blackwell Workstation Edition; remaining
blockers for 8-8 and 8-9 are data coverage and statistical significance rather
than hardware availability.
chapter8/Intuitor/README.md
+1858
View File
File diff suppressed because it is too large Load Diff
chapter8/Intuitor/evaluate_from_cache.py
+328
View File
@@ -0,0 +1,328 @@
#!/usr/bin/env python3
"""
从 lighteval 缓存的 parquet 文件中提取答案并计算 GSM8K 准确率
支持 \\boxed{} 和 #### 两种答案格式
"""
import re
import pandas as pd
import argparse
from pathlib import Path
from typing import Optional
def extract_answer_from_boxed(text: str) -> Optional[str]:
"""从 \\boxed{} 格式中提取答案（同时支持 \\(\\boxed{}\\) 形式）"""
if not text:
return None
# 如果是 bytes，转换为字符串
if isinstance(text, bytes):
text = text.decode('utf-8', errors='ignore')
# 确保是字符串
text = str(text)
# Balanced braces so nested LaTeX like \boxed{\frac{1}{2}} is not truncated.
marker = "\\boxed{"
start = text.find(marker)
if start < 0:
return None
i = start + len(marker)
depth = 1
while i < len(text) and depth:
ch = text[i]
if ch == "{":
depth += 1
elif ch == "}":
depth -= 1
i += 1
if depth != 0:
return None
return text[start + len(marker) : i - 1].strip()
def extract_answer_from_gsm8k_format(text: str) -> Optional[str]:
"""从 #### number 格式中提取答案"""
if not text:
return None
# 如果是 bytes，转换为字符串
if isinstance(text, bytes):
text = text.decode('utf-8', errors='ignore')
# 确保是字符串
text = str(text)
if "####" in text:
parts = text.split("####")
if len(parts) > 1:
return parts[-1].strip()
return None
def _format_normalized_number(num: float) -> str:
if num.is_integer():
return str(int(num))
return str(num)
def normalize_number(text: str) -> Optional[str]:
"""标准化数字格式：去除逗号、空格、LaTeX 符号等"""
if not text:
return None
# 如果是 bytes，转换为字符串
if isinstance(text, bytes):
text = text.decode('utf-8', errors='ignore')
# 确保是字符串
text = str(text)
# Unwrap LaTeX formatting before parsing; the wrapped content may itself be numeric.
cleaned = re.sub(r'\\(?:text|mathrm|mathbf)\s*\{([^}]*)\}', r'\1', text)
cleaned = cleaned.replace("\\$", "").replace("$", "").replace("\\,", "").replace("\\text", "")
cleaned = cleaned.replace(",", "")
# Evaluate \frac{a}{b} before brace stripping (else "\frac{6}{2}" becomes "frac62").
frac = re.search(r'(-)?\s*\\(?:d)?frac\s*\{([^{}]+)\}\s*\{([^{}]+)\}', cleaned)
if frac:
try:
sign = -1.0 if frac.group(1) else 1.0
num_match = re.match(r'\s*(-?\s*\d+(?:\.\d+)?)', frac.group(2))
den_match = re.match(r'\s*(-?\s*\d+(?:\.\d+)?)', frac.group(3))
if not num_match or not den_match:
raise ValueError("fraction component does not start with a number")
num = float(num_match.group(1).replace(" ", ""))
den = float(den_match.group(1).replace(" ", ""))
if den != 0:
return _format_normalized_number(sign * (num / den))
except ValueError:
pass
# Plain a/b before taking the first digit run alone; allow spaces and units.
slash = re.search(r'(-?\s*\d+(?:\.\d+)?)\s*/\s*(-?\s*\d+(?:\.\d+)?)', cleaned)
if slash:
try:
num = float(slash.group(1).replace(" ", ""))
den = float(slash.group(2).replace(" ", ""))
if den != 0:
return _format_normalized_number(num / den)
except ValueError:
pass
# 去除 LaTeX 及货币符号
text = text.replace("\\$", "")
text = text.replace("$", "")
text = text.replace("\\,", "")
text = text.replace("\\text", "")
text = text.replace("{", "").replace("}", "")
# 去除逗号和空格
text = text.replace(",", "").replace(" ", "")
# 提取数字（包括小数和负数）
match = re.search(r'-?\d+\.?\d*', text)
if match:
num_str = match.group(0)
try:
return _format_normalized_number(float(num_str))
except ValueError:
return None
return None
def extract_and_normalize_answer(text: str) -> Optional[str]:
"""从模型输出中提取并标准化答案"""
if not text:
return None
# 如果是 bytes，转换为字符串
if isinstance(text, bytes):
text = text.decode('utf-8', errors='ignore')
# 确保是字符串
text = str(text)
# 先尝试提取 boxed 格式
answer = extract_answer_from_boxed(text)
# 如果没找到，尝试 GSM8K 格式
if not answer:
answer = extract_answer_from_gsm8k_format(text)
# 如果还是没找到，尝试从最后一句话提取数字
if not answer:
# 取最后 200 个字符，避免提取到过程中的数字
last_part = text[-200:] if len(text) > 200 else text
answer = last_part
# 标准化数字格式
return normalize_number(answer)
def load_gsm8k_answers(split: str = "test") -> dict:
"""加载 GSM8K 数据集的金标答案
返回一个字典，键是数据集中的原始索引（0-1318），值是标准化后的答案
"""
try:
from datasets import load_dataset
dataset = load_dataset("gsm8k", "main", split=split)
answers = {}
# 注意：这里的索引是数据集中的顺序索引，不是 sample_id
for idx in range(len(dataset)):
item = dataset[idx]
# GSM8K 答案格式：计算过程\n#### 答案
gold_answer = item["answer"]
# 提取 #### 后面的数字
normalized = extract_answer_from_gsm8k_format(gold_answer)
if normalized:
normalized = normalize_number(normalized)
answers[idx] = normalized
print(f"✅ 加载了 {len(answers)} 个金标答案")
return answers
except ImportError:
print("❌ 错误：需要安装 datasets 库")
print("运行：pip install datasets")
return {}
except Exception as e:
print(f"❌ 加载金标答案时出错: {e}")
return {}
def evaluate_from_parquet(parquet_path: str, verbose: bool = False):
"""从 parquet 文件评测"""
print(f"📂 读取预测结果: {parquet_path}")
df = pd.read_parquet(parquet_path)
print(f"📊 总样本数: {len(df)}")
# 加载金标答案
print("📥 加载 GSM8K 金标答案...")
gold_answers = load_gsm8k_answers()
if not gold_answers:
print("❌ 无法加载金标答案，退出")
return
# 评测
correct = 0
total = 0
errors = []
# 调试：显示前几个 sample_id
if verbose:
print(f"\n前 5 个 sample_id: {df['sample_id'].head().tolist()}")
print(f"金标答案的键范围: {min(gold_answers.keys()) if gold_answers else 'N/A'} - {max(gold_answers.keys()) if gold_answers else 'N/A'}")
for idx, row in df.iterrows():
sample_id = row['sample_id']
sample_data = row['sample']
# 转换 sample_id 为原生 int：parquet 的数值列返回 np.int64，
# 直接放进结果里会让最后的 json.dump 抛
# "Object of type int64 is not JSON serializable"，把 -o 输出截断。
try:
sample_id = int(sample_id)
except (TypeError, ValueError):
if verbose:
print(f"⚠️ 样本 {sample_id}: 无法转换为整数")
continue
# 提取模型输出
text_field = sample_data.get('text', [''])
if isinstance(text_field, list):
model_output = text_field[0] if text_field else ''
else:
model_output = text_field if text_field is not None else ''
# 确保 model_output 是字符串
if isinstance(model_output, bytes):
model_output = model_output.decode('utf-8', errors='ignore')
model_output = str(model_output) if model_output else ''
# 提取并标准化答案
pred_answer = extract_and_normalize_answer(model_output)
gold_answer = gold_answers.get(sample_id)
if gold_answer is None:
if verbose and idx < 5:
print(f"⚠️ 样本 {sample_id}: 找不到金标答案")
continue
total += 1
is_correct = pred_answer == gold_answer
if is_correct:
correct += 1
else:
errors.append({
'sample_id': sample_id,
'predicted': pred_answer,
'gold': gold_answer,
'output': model_output[:200] + "..." if len(model_output) > 200 else model_output
})
if verbose and idx < 5:
print(f"\n样本 {sample_id}:")
print(f" 预测: {pred_answer}")
print(f" 金标: {gold_answer}")
print(f" 正确: {'✅' if is_correct else '❌'}")
# 计算准确率
accuracy = correct / total * 100 if total > 0 else 0
print("\n" + "="*80)
print("📈 评测结果")
print("="*80)
print(f"总样本数: {total}")
print(f"正确数量: {correct}")
print(f"错误数量: {total - correct}")
print(f"准确率: {accuracy:.2f}%")
print("="*80)
# 显示部分错误样本
if errors and verbose:
print("\n❌ 前 10 个错误样本:")
for i, error in enumerate(errors[:10], 1):
print(f"\n{i}. 样本 {error['sample_id']}:")
print(f" 预测: {error['predicted']}")
print(f" 金标: {error['gold']}")
print(f" 输出: {error['output']}")
return {
'total': total,
'correct': correct,
'accuracy': accuracy,
'errors': errors
}
def main():
parser = argparse.ArgumentParser(description='从 lighteval 缓存评测 GSM8K 结果')
parser.add_argument('parquet_file', type=str, help='Parquet 文件路径')
parser.add_argument('-v', '--verbose', action='store_true', help='显示详细信息和错误样本')
parser.add_argument('-o', '--output', type=str, help='保存结果到 JSON 文件')
args = parser.parse_args()
if not Path(args.parquet_file).exists():
print(f"❌ 错误：文件不存在: {args.parquet_file}")
return
results = evaluate_from_parquet(args.parquet_file, verbose=args.verbose)
if args.output and results:
import json
with open(args.output, 'w') as f:
json.dump(results, f, indent=2, ensure_ascii=False)
print(f"\n💾 结果已保存到: {args.output}")
if __name__ == "__main__":
main()
chapter8/Intuitor/test_normalize_negative_currency.py
+31
View File
@@ -0,0 +1,31 @@
"""Regression: negative currency formats and negative fractions must preserve negative sign."""
from evaluate_from_cache import extract_answer_from_gsm8k_format
from evaluate_from_cache import extract_and_normalize_answer, normalize_number
def test_negative_dollar_amount():
assert normalize_number("-$42") == "-42"
def test_negative_latex_dollar_amount():
assert normalize_number(r"-\$42") == "-42"
def test_boxed_negative_dollar_amount():
assert extract_and_normalize_answer(r"\boxed{-\$42}") == "-42"
def test_negative_latex_frac():
assert normalize_number(r"-\frac{6}{2}") == "-3"
assert normalize_number(r"-\dfrac{6}{2}") == "-3"
def test_extract_gsm8k_multiple_hash_markers():
assert extract_answer_from_gsm8k_format("#### step 1 #### 42") == "42"
assert extract_and_normalize_answer("#### step 1 #### 42") == "42"
def test_negative_fraction_with_space():
assert normalize_number("- 1/2") == "-0.5"
assert extract_and_normalize_answer(r"\boxed{- 1/2}") == "-0.5"
chapter8/Intuitor/tests/conftest.py
+9
View File
@@ -0,0 +1,9 @@
"""Test import bootstrap for the Intuitor experiment."""
from pathlib import Path
import sys
EXPERIMENT_ROOT = Path(__file__).resolve().parents[1]
if str(EXPERIMENT_ROOT) not in sys.path:
sys.path.insert(0, str(EXPERIMENT_ROOT))
chapter8/Intuitor/tests/test_boxed_nested_braces.py
+18
View File
@@ -0,0 +1,18 @@
"""Regression: nested braces inside \\boxed{} must not truncate."""
from evaluate_from_cache import extract_answer_from_boxed
def test_boxed_nested_frac():
text = r"The answer is \boxed{\frac{1}{2}}"
assert extract_answer_from_boxed(text) == r"\frac{1}{2}"
def test_boxed_simple_integer():
text = r"Final answer: \boxed{42}"
assert extract_answer_from_boxed(text) == "42"
def test_boxed_deeper_nesting():
text = r"\boxed{\frac{a}{b+c}}"
assert extract_answer_from_boxed(text) == r"\frac{a}{b+c}"
chapter8/Intuitor/tests/test_empty_text_list.py
+22
View File
@@ -0,0 +1,22 @@
"""Regression: empty sample text list must not IndexError."""
def _extract_model_output(sample_data):
text_field = sample_data.get('text', [''])
if isinstance(text_field, list):
return text_field[0] if text_field else ''
return text_field if text_field is not None else ''
def test_empty_text_list():
assert _extract_model_output({"text": []}) == ""
def test_nonempty_text_list():
assert _extract_model_output({"text": ["hello"]}) == "hello"
def test_source_guards_empty_list():
from pathlib import Path
src = (Path(__file__).resolve().parents[1] / "evaluate_from_cache.py").read_text()
assert "text_field[0] if text_field else ''" in src
chapter8/Intuitor/tests/test_normalize_frac.py
+20
View File
@@ -0,0 +1,20 @@
"""Regression: \\frac{a}{b} and a/b must evaluate, not concatenate digit runs."""
from evaluate_from_cache import extract_and_normalize_answer, normalize_number
def test_frac_six_over_two():
assert extract_and_normalize_answer(r"\boxed{\frac{6}{2}}") == "3"
def test_plain_slash_six_over_two():
assert extract_and_normalize_answer(r"\boxed{6/2}") == "3"
def test_frac_one_half():
assert extract_and_normalize_answer(r"\boxed{\frac{1}{2}}") == "0.5"
def test_plain_integer_unchanged():
assert extract_and_normalize_answer(r"\boxed{42}") == "42"
assert normalize_number("1,234") == "1234"
chapter8/Intuitor/tests/test_normalize_frac_formatting.py
+26
View File
@@ -0,0 +1,26 @@
"""Regression: LaTeX fractions and division with formatting/units must evaluate correctly."""
from evaluate_from_cache import extract_and_normalize_answer, normalize_number
def test_frac_thin_space_evaluates():
assert normalize_number(r"\frac{1\,000}{2}") == "500"
assert normalize_number(r"\dfrac{1\,500}{3}") == "500"
assert normalize_number(r"-\frac{1\,000}{2}") == "-500"
def test_frac_text_units_evaluates():
assert normalize_number(r"\frac{100\text{ kg}}{2}") == "50"
assert normalize_number(r"\frac{100}{2\text{ kg}}") == "50"
assert normalize_number(r"\frac{\text{100 kg}}{2}") == "50"
def test_frac_numeric_format_wrappers_evaluate():
assert normalize_number(r"\frac{\mathrm{1,000}}{2}") == "500"
assert normalize_number(r"\frac{\mathbf{6}}{2}") == "3"
def test_slash_division_with_units_and_formatting():
assert normalize_number("6/2 kg") == "3"
assert normalize_number("$6/2$") == "3"
assert normalize_number("1,000 / 2") == "500"
chapter8/MiniMind-pretrain/README.md
+2063
View File
File diff suppressed because it is too large Load Diff
chapter8/MiniMind-pretrain/validation/latest.json
+8
View File
@@ -0,0 +1,8 @@
{
"experiment": "8-3",
"manifest_sha256": "2b99fb95727eba2d71cd1bf7e8a25585856efe0e61a7f02d3ff1bf07751fb9a3",
"run_dir": "validation/runs/exp8-3-training-report-20260731-v1",
"run_id": "exp8-3-training-report-20260731-v1",
"schema_version": "exp8-3-latest-v1",
"status": "passed"
}
chapter8/MiniMind-pretrain/validation/latest_vlm.json
+8
View File
@@ -0,0 +1,8 @@
{
"experiment": "8-4",
"manifest_sha256": "f984d549d8172528516d91800871df3dcd6b67f61729314b1371aa8022fabbf0",
"run_dir": "validation/runs/exp8-4-training-report-20260731-v1",
"run_id": "exp8-4-training-report-20260731-v1",
"schema_version": "exp8-4-latest-v1",
"status": "passed"
}
chapter8/MiniMind-pretrain/validation/run_training_report_audit.py
+760
View File
@@ -0,0 +1,760 @@
#!/usr/bin/env python3
"""Build checkpoint-free retained training evidence for Experiment 8-3.
The book already contains the author's historical six-cell evaluation report:
original versus QK-Norm + Muon at pretrain, SFT, and DPO. This program does
not pretend to rerun the GPU training job. It extracts every saved generation,
submits a preregistered stage-balanced subset to an arm-blind external judge,
and binds the raw report, judge receipts, immutable future-reproduction source
and dataset revisions, environment lock, findings, and limitations into a
content-hashed evidence package. Checkpoints are intentionally not published
and are not an acceptance artifact for book training experiments.
"""
from __future__ import annotations
import argparse
import concurrent.futures
import hashlib
import json
import os
import random
import re
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
EXPERIMENT_DIR = HERE.parent
REPO_ROOT = EXPERIMENT_DIR.parents[1]
REPORT_PATH = EXPERIMENT_DIR / "README.md"
RUNS_DIR = HERE / "runs"
LATEST_PATH = HERE / "latest.json"
DEFAULT_RUN_ID = "exp8-3-training-report-20260731-v1"
DEFAULT_ENDPOINT = "https://ark.cn-beijing.volces.com/api/v3/chat/completions"
DEFAULT_MODEL = "doubao-seed-1-6-250615"
BLIND_SEED = 730731
SOURCE_REVISION = "8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795"
DATASET_REVISION = "84983ed4dec7836d240577760c1d6be5d4cabcf9"
SOURCE_FILES = {
"model/model_minimind.py": "2d33988711c704be6a22c4c61489b23106a2340a7cb8b97ebe3e40f30819cbb0",
"model/tokenizer.json": "e489029175fb3f94b8211a120a72a2ee41a664db65b828d077c7bde989c845a9",
"model/tokenizer_config.json": "190cc4738bac3b6f6b563376019c581b320fdb0260a03b9d5ab806296c8c6bb8",
"dataset/lm_dataset.py": "213726b1781289548784220b2b2db48fe97d84f3b67ac0cd70186cbbcb7b5d2c",
"trainer/muon.py": "00c2c6a225edeb55433df0724c3c74f6ff98ac4b2cc73c4aafcff686824f6267",
"trainer/train_pretrain.py": "ddd122645a9f1043bc8dac69a81ac51d2df95df8745d25faed7963d38fedc328",
"trainer/train_pretrain_muon.py": "fc83d07754ec3a8c156b6b8bfc0fd4326edecb72efabc5e08ae4ff5e3a7029bc",
"trainer/train_full_sft.py": "a57422f1df80bf2867f31f3b4a646a92ac7f66729e98a3b32cd1ec4d6780cb8b",
"trainer/train_full_sft_muon.py": "acd0b7db5b1d8b25d3c3103f92d68a7d381f9322a1b33bbec34be3d005930bad",
"trainer/train_dpo.py": "97f2c31cc8bc21a777e2efcb5e2fa35a49e4e9e3698db120148f8a0b2f678449",
"eval_model.py": "43930a4b55048a4a3ffa17eb78ae67d59582d639aa9365f0bbf41ba149128af8",
"requirements.txt": "23f4cea09281765eec7cf03e28231425638e8e42580f418781fed166b75af968",
}
DATASET_FILES = {
"pretrain_hq.jsonl": {
"lfs_sha256": "9801b0d2210c61c2e4bc130f6dc4b3c870698a88d04af8f103c23dd5f0ce2440",
"bytes": 1_669_750_047,
},
"sft_512.jsonl": {
"lfs_sha256": "053b7d09574e48a86232e929211434ff9e5016c6ed13312e63687dd52edcbebf",
"bytes": 7_531_517_862,
},
"dpo.jsonl": {
"lfs_sha256": "ee934a8a455ccc99d1334d63e1254dd1d64f497fd067cfcbb71e3043f5b46768",
"bytes": 53_653_322,
},
}
ARMS = ("original", "qk_norm_muon")
STAGES = ("pretrain", "sft", "dpo")
EXPECTED_COUNTS = {
("original", "pretrain"): 7,
("original", "sft"): 8,
("original", "dpo"): 9,
("qk_norm_muon", "pretrain"): 7,
("qk_norm_muon", "sft"): 9,
("qk_norm_muon", "dpo"): 9,
}
SELECTED_CASES = (
{"case_id": 1, "stage": "pretrain", "keyword": "highest mountain", "task": "Continue the prompt by identifying the highest mountain in the world accurately."},
{"case_id": 2, "stage": "pretrain", "keyword": "carbon dioxide", "task": "Continue the prompt with an accurate statement about carbon dioxide in air."},
{"case_id": 3, "stage": "sft", "keyword": "speed of light", "task": "Explain the physical concept of the speed of light in detail."},
{"case_id": 4, "stage": "sft", "keyword": "how to understand chatgpt", "task": "Explain what ChatGPT is and how it works."},
{"case_id": 5, "stage": "sft", "keyword": "history of the united states", "task": "Introduce the history of the United States."},
{"case_id": 6, "stage": "dpo", "keyword": "speed of light", "task": "Explain the physical concept of the speed of light in detail."},
{"case_id": 7, "stage": "dpo", "keyword": "how to understand chatgpt", "task": "Explain what ChatGPT is and how it works."},
{"case_id": 8, "stage": "dpo", "keyword": "history of the united states", "task": "Introduce the history of the United States."},
)
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def canonical_json_bytes(value: Any) -> bytes:
return (json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode()
def write_json(path: Path, value: Any) -> None:
path.write_bytes(canonical_json_bytes(value))
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def _terminal_transcript(section: str) -> tuple[str, str]:
"""Return the terminal header and prompt/output transcript from one cell.
The historical Markdown has one stray closing fence immediately before the
improved-SFT Lu Xun answer. Treat fences as presentation markup rather
than semantic delimiters so that the retained answer is not silently lost.
"""
prompt_position = section.find("👶:")
if prompt_position < 0:
raise ValueError("model section has no user prompt marker")
header = section[:prompt_position].replace("```", "").strip()
transcript = section[prompt_position:]
analysis = re.search(r"(?m)^\*\*[^\n]*Analysis[^\n]*\*\*:?\s*$", transcript)
if analysis:
transcript = transcript[: analysis.start()]
transcript = re.sub(r"(?m)^```\s*$", "", transcript).strip()
return header, transcript
def _parse_pairs(transcript: str) -> list[dict[str, str]]:
pattern = re.compile(
r"^👶:\s*(.*?)\n🤖️:\s*(.*?)(?=\n(?:\s*\n)*👶:|\Z)",
flags=re.DOTALL | re.MULTILINE,
)
pairs = []
for match in pattern.finditer(transcript):
prompt = match.group(1).strip()
output = match.group(2).strip()
if not prompt or not output:
raise ValueError("empty prompt or output in historical transcript")
pairs.append({"prompt": prompt, "output": output})
return pairs
def parse_retained_outputs(report_path: Path = REPORT_PATH) -> dict[str, Any]:
"""Extract all six historical LLM evaluation cells from the bilingual report."""
text = report_path.read_text(encoding="utf-8")
start = text.index("## Language Model Training Results Analysis")
end = text.index("# Analysis of Vision-Language Model Training Results", start)
llm_text = text[start:end]
arm_markers = {
"original": "## Without Muon Optimizer (Original Architecture)",
"qk_norm_muon": "## With Muon Optimizer and QK Norm (Improved Architecture)",
}
cells: list[dict[str, Any]] = []
for arm_index, arm in enumerate(ARMS):
arm_start = llm_text.index(arm_markers[arm])
arm_end = (
llm_text.index(arm_markers[ARMS[arm_index + 1]], arm_start)
if arm_index + 1 < len(ARMS)
else len(llm_text)
)
arm_text = llm_text[arm_start:arm_end]
for stage_index, stage in enumerate(STAGES):
heading = {"pretrain": "### Pretrain Model", "sft": "### SFT Model", "dpo": "### DPO Model"}[stage]
cell_start = arm_text.index(heading)
next_positions = [
arm_text.find(next_heading, cell_start + len(heading))
for next_heading in ("### Pretrain Model", "### SFT Model", "### DPO Model")
]
next_positions = [position for position in next_positions if position >= 0]
cell_end = min(next_positions) if next_positions else len(arm_text)
header, transcript = _terminal_transcript(arm_text[cell_start:cell_end])
pairs = _parse_pairs(transcript)
expected = EXPECTED_COUNTS[(arm, stage)]
if len(pairs) != expected:
raise ValueError(f"{arm}/{stage}: expected {expected} pairs, found {len(pairs)}")
cells.append(
{
"arm": arm,
"stage": stage,
"terminal_header": header,
"pair_count": len(pairs),
"pairs": pairs,
}
)
return {
"schema_version": "exp8-3-retained-outputs-v1",
"experiment": "8-3",
"source_report": str(report_path.relative_to(REPO_ROOT)),
"source_report_sha256": sha256_file(report_path),
"arms": list(ARMS),
"stages": list(STAGES),
"cell_count": len(cells),
"output_count": sum(cell["pair_count"] for cell in cells),
"cells": cells,
}
def _find_pair(retained: dict[str, Any], arm: str, stage: str, keyword: str) -> dict[str, str]:
cell = next(cell for cell in retained["cells"] if cell["arm"] == arm and cell["stage"] == stage)
matches = [pair for pair in cell["pairs"] if keyword in pair["prompt"].lower()]
if len(matches) != 1:
raise ValueError(f"{arm}/{stage}/{keyword}: expected one prompt, found {len(matches)}")
return matches[0]
def selected_comparisons(retained: dict[str, Any]) -> list[dict[str, Any]]:
comparisons = []
for case in SELECTED_CASES:
rows = {
arm: _find_pair(retained, arm, case["stage"], case["keyword"])
for arm in ARMS
}
comparisons.append({**case, "arms": rows})
return comparisons
def blind_mapping(case_id: int) -> dict[str, str]:
arms = list(ARMS)
random.Random(BLIND_SEED + case_id).shuffle(arms)
return dict(zip(("A", "B"), arms, strict=True))
def judge_payload(comparison: dict[str, Any], mapping: dict[str, str], model: str) -> dict[str, Any]:
candidates = {
label: {
"historical_prompt": comparison["arms"][arm]["prompt"],
"historical_output": comparison["arms"][arm]["output"],
}
for label, arm in mapping.items()
}
required = {
"case_id": comparison["case_id"],
"candidates": {
label: {
"language_fluency": "number 0-5",
"instruction_following": "number 0-5",
"factuality": "number 0-5",
"factual_errors": ["specific material errors; empty only if none"],
"rationale": "brief evidence-based explanation",
}
for label in ("A", "B")
},
"winner": "A, B, or tie",
}
return {
"model": model,
"temperature": 0,
"response_format": {"type": "json_object"},
"messages": [
{
"role": "system",
"content": (
"You are an independent evaluator of two anonymous small-language-model outputs. "
"Do not infer model identity, architecture, optimizer, or training stage. Score only "
"the supplied text. Penalize hallucinations, unsafe medical specificity, repetition, "
"and non-answers. Return one JSON object with exactly the requested fields."
),
},
{
"role": "user",
"content": json.dumps(
{
"case_id": comparison["case_id"],
"task": comparison["task"],
"rubric": {
"language_fluency": "0 unreadable; 3 understandable with defects; 5 coherent and natural",
"instruction_following": "0 non-answer; 3 partial; 5 directly and fully answers",
"factuality": "0 dominated by falsehoods; 3 mixed/minor errors; 5 no material error",
},
"candidates": candidates,
"required_json_shape": required,
},
ensure_ascii=False,
sort_keys=True,
),
},
],
}
def extract_json_object(content: str) -> dict[str, Any]:
stripped = content.strip()
if stripped.startswith("```"):
stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
stripped = re.sub(r"\s*```$", "", stripped)
parsed = json.loads(stripped)
if not isinstance(parsed, dict):
raise ValueError("judge content must decode to an object")
return parsed
def validate_judgment(judgment: dict[str, Any], case_id: int) -> None:
if str(judgment.get("case_id")) != str(case_id):
raise ValueError(
f"judge returned the wrong case_id: expected {case_id}, got {judgment.get('case_id')!r}"
)
candidates = judgment.get("candidates")
if not isinstance(candidates, dict) or set(candidates) != {"A", "B"}:
raise ValueError("judge must score A and B exactly")
for label in ("A", "B"):
row = candidates[label]
if not isinstance(row, dict):
raise ValueError(f"candidate {label} score must be an object")
for metric in ("language_fluency", "instruction_following", "factuality"):
score = row.get(metric)
if not isinstance(score, (int, float)) or isinstance(score, bool) or not 0 <= score <= 5:
raise ValueError(f"candidate {label} has invalid {metric}: {score!r}")
if not isinstance(row.get("factual_errors"), list):
raise ValueError(f"candidate {label} factual_errors must be a list")
if not isinstance(row.get("rationale"), str) or not row["rationale"].strip():
raise ValueError(f"candidate {label} rationale is missing")
if judgment.get("winner") not in {"A", "B", "tie"}:
raise ValueError("judge winner must be A, B, or tie")
def call_judge(
comparison: dict[str, Any], *, endpoint: str, model: str, api_key: str, timeout: float
) -> dict[str, Any]:
mapping = blind_mapping(comparison["case_id"])
payload = judge_payload(comparison, mapping, model)
request = urllib.request.Request(
endpoint,
data=json.dumps(payload, ensure_ascii=False).encode(),
method="POST",
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
)
started = time.perf_counter()
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
raw_body = response.read()
http_status = response.status
except urllib.error.HTTPError as exc:
body = exc.read().decode(errors="replace")
raise RuntimeError(f"judge HTTP {exc.code}: {body[:500]}") from exc
latency_ms = round((time.perf_counter() - started) * 1000, 3)
raw_response = json.loads(raw_body)
content = raw_response["choices"][0]["message"]["content"]
judgment = extract_json_object(content)
validate_judgment(judgment, comparison["case_id"])
response_id = raw_response.get("id")
usage = raw_response.get("usage")
if not isinstance(response_id, str) or not response_id:
raise ValueError("judge response has no response ID")
if not isinstance(usage, dict) or not isinstance(usage.get("total_tokens"), int):
raise ValueError("judge response has no complete usage object")
return {
"case_id": comparison["case_id"],
"stage": comparison["stage"],
"keyword": comparison["keyword"],
"task": comparison["task"],
"provider": "ark",
"endpoint": endpoint,
"credential_env": "ARK_API_KEY",
"credential_headers_retained": False,
"blind_seed": BLIND_SEED,
"blind_map": mapping,
"request": payload,
"http_status": http_status,
"response": raw_response,
"response_id": response_id,
"usage": usage,
"latency_ms": latency_ms,
"judgment": judgment,
}
def reproduction_contract() -> dict[str, Any]:
return {
"schema_version": "exp8-3-reproduction-contract-v1",
"experiment": "8-3",
"historical_evidence_boundary": {
"historical_training_executed": True,
"six_historical_evaluation_transcripts_retained": True,
"historical_source_revision_retained": False,
"historical_dataset_hashes_retained": False,
"historical_checkpoint_hashes_retained": False,
"historical_stepwise_training_logs_retained": False,
"claim": (
"The author's retained report supports that original and QK-Norm+Muon 104.03M models "
"were evaluated after pretrain, SFT, and DPO. It does not establish byte identity of the "
"historical checkpoints, datasets, source checkout, or every loss point."
),
},
"future_reproduction": {
"source": {
"repository": "bojieli/minimind",
"revision": SOURCE_REVISION,
"selected_at": "2026-07-31",
"not_claimed_as_historical_revision": True,
"file_sha256": SOURCE_FILES,
},
"dataset": {
"repository": "jingyaogong/minimind_dataset",
"revision": DATASET_REVISION,
"selected_at": "2026-07-31",
"not_claimed_as_historical_revision": True,
"files": DATASET_FILES,
},
"commands": {
"original_pretrain": "torchrun --nproc_per_node=8 trainer/train_pretrain.py --epochs 10 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/pretrain_hq.jsonl --use_wandb",
"improved_pretrain": "torchrun --nproc_per_node=8 trainer/train_pretrain_muon.py --epochs 10 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/pretrain_hq.jsonl --use_wandb",
"original_sft": "torchrun --nproc_per_node=8 trainer/train_full_sft.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/sft_512.jsonl --use_wandb",
"improved_sft": "torchrun --nproc_per_node=8 trainer/train_full_sft_muon.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/sft_512.jsonl --use_wandb",
"original_dpo": "torchrun --nproc_per_node=8 trainer/train_dpo.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --init_from out/full_sft_768.pth --data_path dataset/dpo.jsonl --use_wandb",
"improved_dpo": "torchrun --nproc_per_node=8 trainer/train_dpo.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --init_from out/full_sft_muon_768.pth --data_path dataset/dpo.jsonl --use_wandb",
},
"environment": {
"book_pyproject": "pyproject.toml",
"book_lock": "uv.lock",
"install": "uv sync --locked --python 3.12 --extra ch7 --extra dev",
"boundary": (
"The book lock freezes a future software environment. The pinned upstream requirements "
"file itself is unversioned, and the future GPU/CUDA stack has not been exercised here."
),
},
},
"model_and_training": {
"reported_parameter_count_millions": 104.03,
"architecture": {"hidden_size": 768, "layers": 16, "sequence_length": 512},
"stages": list(STAGES),
"arms": list(ARMS),
"source_verified_mechanisms": {
"qk_norm_before_rope": True,
"muon_for_two_dimensional_non_embedding_weights": True,
"adamw_for_embeddings_norms_and_lm_head": True,
"dpo_uses_adamw_from_arm_specific_sft_checkpoint": True,
},
"reported_scalars_without_stepwise_logs": {
"steps_to_loss_3_original": 36,
"steps_to_loss_3_qk_norm_muon": 12,
"final_loss_original": 2.0,
"final_loss_qk_norm_muon": 1.7,
"eight_rtx_4090_pretrain_hours": 6,
"eight_rtx_4090_sft_hours": 8,
},
},
"checkpoint_policy": {
"distributed_with_book": False,
"acceptance_artifact": False,
"required_artifact": "reproducible evidence-backed training report",
"reason": "Training checkpoints are intentionally not distributed to readers.",
},
}
def mean(values: list[float]) -> float:
return round(sum(values) / len(values), 4)
def summarize(retained: dict[str, Any], receipts: list[dict[str, Any]], contract: dict[str, Any]) -> dict[str, Any]:
metrics = ("language_fluency", "instruction_following", "factuality")
rows: dict[int, dict[str, dict[str, Any]]] = {}
wins = {"original": 0, "qk_norm_muon": 0, "tie": 0}
for receipt in receipts:
reverse = receipt["blind_map"]
rows[receipt["case_id"]] = {
reverse[label]: score for label, score in receipt["judgment"]["candidates"].items()
}
winner = receipt["judgment"]["winner"]
wins["tie" if winner == "tie" else reverse[winner]] += 1
arm_averages = {}
stage_averages = {}
for arm in ARMS:
arm_scores = [rows[case["case_id"]][arm] for case in SELECTED_CASES]
arm_averages[arm] = {
metric: mean([float(row[metric]) for row in arm_scores]) for metric in metrics
}
arm_averages[arm]["overall"] = mean(
[float(row[metric]) for row in arm_scores for metric in metrics]
)
stage_averages[arm] = {}
for stage in STAGES:
stage_scores = [
rows[case["case_id"]][arm] for case in SELECTED_CASES if case["stage"] == stage
]
stage_averages[arm][stage] = {
metric: mean([float(row[metric]) for row in stage_scores]) for metric in metrics
}
stage_averages[arm][stage]["overall"] = mean(
[float(row[metric]) for row in stage_scores for metric in metrics]
)
findings = {
"blind_judge_overall_delta_qk_norm_muon_minus_original": round(
arm_averages["qk_norm_muon"]["overall"] - arm_averages["original"]["overall"], 4
),
"blind_judge_prefers_qk_norm_muon_overall": (
arm_averages["qk_norm_muon"]["overall"] > arm_averages["original"]["overall"]
),
"wins": wins,
"reported_loss_comparison_retained_but_not_independently_recomputed": True,
}
acceptance = {
"raw_historical_report_hashed": bool(retained["source_report_sha256"]),
"all_six_arm_stage_cells_retained": retained["cell_count"] == 6,
"all_expected_outputs_retained": retained["output_count"] == sum(EXPECTED_COUNTS.values()),
"pretrain_sft_and_dpo_compared": set(retained["stages"]) == set(STAGES),
"original_and_qk_norm_muon_compared": set(retained["arms"]) == set(ARMS),
"eight_stage_balanced_blind_judgments": len(receipts) == len(SELECTED_CASES),
"raw_judge_requests_responses_ids_usage_latency_retained": all(
receipt["response_id"]
and receipt["usage"].get("total_tokens", 0) > 0
and receipt["latency_ms"] > 0
for receipt in receipts
),
"immutable_source_revision_and_file_hashes_frozen": bool(SOURCE_REVISION and SOURCE_FILES),
"immutable_dataset_revision_lfs_hashes_and_sizes_frozen": bool(DATASET_REVISION and DATASET_FILES),
"future_reproduction_commands_declared": len(contract["future_reproduction"]["commands"]) == 6,
"historical_provenance_limitations_explicit": (
contract["historical_evidence_boundary"]["historical_checkpoint_hashes_retained"] is False
and contract["historical_evidence_boundary"]["historical_stepwise_training_logs_retained"] is False
),
"reported_loss_claims_qualified": findings[
"reported_loss_comparison_retained_but_not_independently_recomputed"
],
"checkpoints_not_an_acceptance_artifact": (
contract["checkpoint_policy"]["acceptance_artifact"] is False
),
}
passed = all(acceptance.values())
return {
"schema_version": "exp8-3-summary-v1",
"experiment": "8-3",
"status": "passed" if passed else "failed",
"judge": {
"provider": "ark",
"model": receipts[0]["request"]["model"],
"calls": len(receipts),
"response_ids": [receipt["response_id"] for receipt in receipts],
"total_tokens": sum(receipt["usage"]["total_tokens"] for receipt in receipts),
"total_latency_ms": round(sum(receipt["latency_ms"] for receipt in receipts), 3),
"blind_seed": BLIND_SEED,
},
"retained": {
"cells": retained["cell_count"],
"outputs": retained["output_count"],
"selected_comparisons": len(receipts),
},
"arm_averages": arm_averages,
"stage_averages": stage_averages,
"per_case_arm_scores": rows,
"scientific_findings": findings,
"acceptance": {**acceptance, "passed": passed},
"limitations": [
"Historical checkpoints are intentionally not distributed and were not recreated in this audit.",
"The historical source revision, dataset byte identities, RNG state, and stepwise loss logs were not retained.",
"Frozen source/data revisions and the book lock define a future reproduction contract, not historical provenance.",
"The independent judge covers eight preregistered comparisons; all other retained outputs remain available for inspection.",
"The historical outputs are English translations in a bilingual report, so translation may affect the judge scores.",
],
}
def render_report(summary: dict[str, Any]) -> str:
averages = summary["arm_averages"]
findings = summary["scientific_findings"]
return "\n".join(
[
"# Experiment 8-3 retained-training-report audit",
"",
"## Result",
"",
f"Status: **{summary['status']}**. The historical report retains "
f"{summary['retained']['outputs']} outputs across the original and QK-Norm + Muon arms "
"after pretrain, SFT, and DPO. Eight preregistered arm-blind comparisons were judged "
"from raw retained text by an independent ARK model.",
"",
"| Arm | Fluency | Instruction | Factuality | Overall |",
"| --- | ---: | ---: | ---: | ---: |",
f"| Original | {averages['original']['language_fluency']:.4f} | "
f"{averages['original']['instruction_following']:.4f} | "
f"{averages['original']['factuality']:.4f} | {averages['original']['overall']:.4f} |",
f"| QK-Norm + Muon | {averages['qk_norm_muon']['language_fluency']:.4f} | "
f"{averages['qk_norm_muon']['instruction_following']:.4f} | "
f"{averages['qk_norm_muon']['factuality']:.4f} | {averages['qk_norm_muon']['overall']:.4f} |",
"",
f"Observed blind-judge overall delta: **{findings['blind_judge_overall_delta_qk_norm_muon_minus_original']:+.4f}**. "
f"Pairwise decisions: {findings['wins']}.",
"",
"The report's loss claims (3.0 reached at 36 versus 12 reported steps; final loss 2.0 "
"versus 1.7) are retained as author-reported observations, not independently recomputed "
"measurements, because the historical stepwise logs were not preserved.",
"",
"## Provenance and reproduction boundary",
"",
"`reproduction_contract.json` freezes the MiniMind source revision, hashes the relevant "
"source files, freezes a dataset revision with the three Git-LFS object hashes and sizes, "
"and records all six future reproduction commands. These pins were selected for future "
"reproduction and are not represented as the exact historical checkout.",
"",
"Training checkpoints remain local by book policy and are not an acceptance artifact. "
"The accepted artifact is this content-hashed training report, its raw retained outputs, "
"raw independent-judge receipts, and explicit limitations.",
"",
]
)
def input_record(path: Path) -> dict[str, Any]:
return {
"path": str(path.relative_to(REPO_ROOT)),
"sha256": sha256_file(path),
"bytes": path.stat().st_size,
}
def artifact_record(path: Path, run_dir: Path) -> dict[str, Any]:
return {
"path": str(path.relative_to(run_dir)),
"sha256": sha256_file(path),
"bytes": path.stat().st_size,
}
def build_manifest(run_id: str, run_dir: Path, summary: dict[str, Any]) -> dict[str, Any]:
inputs = [
input_record(REPORT_PATH),
input_record(REPO_ROOT / "pyproject.toml"),
input_record(REPO_ROOT / "uv.lock"),
input_record(HERE / "run_training_report_audit.py"),
input_record(HERE / "validate_evidence.py"),
]
artifact_paths = [
run_dir / "retained_outputs.json",
run_dir / "reproduction_contract.json",
run_dir / "judge_receipts.json",
run_dir / "summary.json",
run_dir / "report.md",
]
return {
"schema_version": "exp8-3-manifest-v1",
"experiment": "8-3",
"run_id": run_id,
"created_at": utc_now(),
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"inputs": inputs,
"artifacts": [artifact_record(path, run_dir) for path in artifact_paths],
"acceptance": summary["acceptance"],
"checkpoint_policy": "not distributed; not an acceptance artifact",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--run-id", default=DEFAULT_RUN_ID)
parser.add_argument("--endpoint", default=os.getenv("ARK_BASE_URL", DEFAULT_ENDPOINT))
parser.add_argument("--model", default=os.getenv("ARK_MODEL", DEFAULT_MODEL))
parser.add_argument("--api-key-env", default="ARK_API_KEY")
parser.add_argument("--timeout", type=float, default=180.0)
parser.add_argument("--concurrency", type=int, default=8)
parser.add_argument(
"--refresh-manifest",
action="store_true",
help="Rehash an existing run after source-only corrections; makes no provider call.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
if not re.fullmatch(r"[A-Za-z0-9._-]+", args.run_id):
raise SystemExit("run ID may contain only letters, digits, dot, underscore, and hyphen")
run_dir = RUNS_DIR / args.run_id
if args.refresh_manifest:
if not run_dir.is_dir():
raise SystemExit(f"cannot refresh missing run: {run_dir}")
stored_retained = json.loads(
(run_dir / "retained_outputs.json").read_text(encoding="utf-8")
)
current_retained = parse_retained_outputs()
# Documentation around the raw transcripts may change, but a refresh
# must never silently replace the generations that were judged.
if stored_retained.get("cells") != current_retained.get("cells"):
raise SystemExit(
"refusing manifest refresh because retained historical outputs changed"
)
stored_retained["source_report_sha256"] = current_retained[
"source_report_sha256"
]
write_json(run_dir / "retained_outputs.json", stored_retained)
summary = json.loads((run_dir / "summary.json").read_text(encoding="utf-8"))
write_json(run_dir / "manifest.json", build_manifest(args.run_id, run_dir, summary))
latest = {
"schema_version": "exp8-3-latest-v1",
"experiment": "8-3",
"run_id": args.run_id,
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"manifest_sha256": sha256_file(run_dir / "manifest.json"),
}
write_json(LATEST_PATH, latest)
print(json.dumps(latest, indent=2, sort_keys=True))
return 0
if run_dir.exists():
raise SystemExit(f"refusing to overwrite existing run: {run_dir}")
api_key = os.getenv(args.api_key_env)
if not api_key:
raise SystemExit(f"missing required credential environment variable: {args.api_key_env}")
retained = parse_retained_outputs()
comparisons = selected_comparisons(retained)
with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as pool:
futures = {
comparison["case_id"]: pool.submit(
call_judge,
comparison,
endpoint=args.endpoint,
model=args.model,
api_key=api_key,
timeout=args.timeout,
)
for comparison in comparisons
}
receipts = [futures[case_id].result() for case_id in sorted(futures)]
contract = reproduction_contract()
summary = summarize(retained, receipts, contract)
run_dir.mkdir(parents=True)
write_json(run_dir / "retained_outputs.json", retained)
write_json(
run_dir / "judge_receipts.json",
{
"schema_version": "exp8-3-judge-receipts-v1",
"experiment": "8-3",
"credential_headers_retained": False,
"calls": receipts,
},
)
write_json(run_dir / "reproduction_contract.json", contract)
write_json(run_dir / "summary.json", summary)
(run_dir / "report.md").write_text(render_report(summary), encoding="utf-8")
write_json(run_dir / "manifest.json", build_manifest(args.run_id, run_dir, summary))
latest = {
"schema_version": "exp8-3-latest-v1",
"experiment": "8-3",
"run_id": args.run_id,
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"manifest_sha256": sha256_file(run_dir / "manifest.json"),
}
write_json(LATEST_PATH, latest)
print(json.dumps(latest, indent=2, sort_keys=True))
return 0 if summary["status"] == "passed" else 1
if __name__ == "__main__":
raise SystemExit(main())
chapter8/MiniMind-pretrain/validation/run_vlm_training_report_audit.py
Executable
+922
View File
@@ -0,0 +1,922 @@
#!/usr/bin/env python3
"""Build checkpoint-free retained training evidence for Experiment 8-4.
The book contains 64 historical MiniMind-V image descriptions: eight model
configurations evaluated on the same eight images. This program extracts all
of them, asks a real image-capable model to judge every anonymous candidate
against the corresponding source image, and writes raw credential-free
requests/responses plus a fail-closed, content-hashed reproduction package.
It deliberately does not claim to rerun the historical GPU jobs. Historical
checkpoints are intentionally not distributed; the accepted artifact is a
reproducible training report with explicit provenance limits.
"""
from __future__ import annotations
import argparse
import base64
import concurrent.futures
import hashlib
import json
import mimetypes
import os
import random
import re
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
EXPERIMENT_DIR = HERE.parent
REPO_ROOT = EXPERIMENT_DIR.parents[1]
REPORT_PATH = EXPERIMENT_DIR / "README.md"
RUNS_DIR = HERE / "runs"
LATEST_PATH = HERE / "latest_vlm.json"
DEFAULT_RUN_ID = "exp8-4-training-report-20260731-v1"
DEFAULT_ENDPOINT = "https://ark.cn-beijing.volces.com/api/v3/chat/completions"
DEFAULT_MODEL = "doubao-seed-1-6-250615"
BLIND_SEED = 740731
ORIGINAL_VLM_REVISION = "765908051d0837d60cecfb93f8390334e2e55f1e"
IMPROVED_VLM_REVISION = "ead791c530fa5f9a3549dbfe9e11ec732d18d2e5"
ORIGINAL_LLM_REVISION = "6d160ea20b98324632c4447ee63ec7cfa9becd20"
IMPROVED_LLM_REVISION = "8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795"
DATASET_REVISION = "ac9d03a3fd26a2d8e74bda374d9a2ddba49e4c1b"
CLIP_REVISION = "57c216476eefef5ab752ec549e440a49ae4ae5f3"
ORIGINAL_VLM_FILES = {
"trainer/train_pretrain_vlm.py": "4d30d54a940ae2eced204971cc03aafb3eb41a5c84c9f033cfdf162e63924a4d",
"trainer/train_sft_vlm.py": "8e3b920a6a135eb126bbeea07e2db748729cdd86050925b282a80537bc324e5f",
"eval_vlm.py": "9d883e4adbab0a7b88fd0cb9034132559a365387ec273ac4811cdd5ad28d5cda",
"model/model_minimind.py": "105429e93dcbe87145264d72d46a6add7639666036e999628c76ae50582507dc",
"model/model_vlm.py": "4ee42b298db68f30fbfa06d7686aa375d41a697628c770d0a134bca40ca9ea80",
"dataset/lm_dataset.py": "df20d57460d2845841ddf2e0faced1af1f7ec169e7fda3cd50fe3b2854288a92",
"model/tokenizer.json": "d98595c6aef70d95f72748582fb9b4f53d76dd58c1ae1dd702ad7c84e1caf5e4",
"model/tokenizer_config.json": "dbbdb7eea33aba5c2608471494c93f650a2cf46fbe4a7489e531537ddadee746",
"requirements.txt": "a9bddf49d3ccbc9f8a2508ea039aebc0b996dccb0d3618d1b119af53a5d49869",
}
IMPROVED_VLM_FILES = {
"trainer/train_pretrain_vlm_muon.py": "f39af354c588747d9d5e522c9374a7f59a35d57aa649da74957da67d95d25bc6",
"trainer/train_sft_vlm_muon.py": "1fd56b3e8bed2714b4d10ceba5d57ada0b95d00dfbfb514481498fef0c0dd03d",
"trainer/muon.py": "00c2c6a225edeb55433df0724c3c74f6ff98ac4b2cc73c4aafcff686824f6267",
"eval_vlm.py": "9d883e4adbab0a7b88fd0cb9034132559a365387ec273ac4811cdd5ad28d5cda",
"model/model_minimind.py": "4771bc4b2ac367a6e6415c42c30bcdb54bec0397708f87de3c390042680b1e9e",
"model/model_vlm.py": "4ee42b298db68f30fbfa06d7686aa375d41a697628c770d0a134bca40ca9ea80",
"dataset/lm_dataset.py": "df20d57460d2845841ddf2e0faced1af1f7ec169e7fda3cd50fe3b2854288a92",
"model/tokenizer.json": "e489029175fb3f94b8211a120a72a2ee41a664db65b828d077c7bde989c845a9",
"model/tokenizer_config.json": "190cc4738bac3b6f6b563376019c581b320fdb0260a03b9d5ab806296c8c6bb8",
"requirements.txt": "a9bddf49d3ccbc9f8a2508ea039aebc0b996dccb0d3618d1b119af53a5d49869",
}
ORIGINAL_LLM_FILES = {
"model/model_minimind.py": "7cb069cb0cb0dfa123cf11ea394d0001270bc683c0a2dfe4120fc3b861ffc0a4",
"trainer/train_pretrain.py": "ddd122645a9f1043bc8dac69a81ac51d2df95df8745d25faed7963d38fedc328",
"trainer/train_full_sft.py": "a57422f1df80bf2867f31f3b4a646a92ac7f66729e98a3b32cd1ec4d6780cb8b",
"trainer/train_dpo.py": "5e556a3089e43681638cdbf5adafb9d085bb1de5e4ea8da3ee522dfae02e3599",
"eval_model.py": "b9f7ea9d7f517551362bbf2da8f1de006b8c734bcba774b2be752bc63cc4349d",
}
IMPROVED_LLM_FILES = {
"model/model_minimind.py": "2d33988711c704be6a22c4c61489b23106a2340a7cb8b97ebe3e40f30819cbb0",
"trainer/train_pretrain_muon.py": "fc83d07754ec3a8c156b6b8bfc0fd4326edecb72efabc5e08ae4ff5e3a7029bc",
"trainer/train_full_sft_muon.py": "acd0b7db5b1d8b25d3c3103f92d68a7d381f9322a1b33bbec34be3d005930bad",
"trainer/train_dpo.py": "97f2c31cc8bc21a777e2efcb5e2fa35a49e4e9e3698db120148f8a0b2f678449",
"eval_model.py": "43930a4b55048a4a3ffa17eb78ae67d59582d639aa9365f0bbf41ba149128af8",
}
DATASET_FILES = {
"pretrain_data.jsonl": {
"lfs_sha256": "abc9f2ba44190646692fbe7e2b49c366c5045490989fb32d2c5e960dd0ee10e4",
"bytes": 134315765,
},
"pretrain_images.zip": {
"lfs_sha256": "64d56cee145bed75bc7f94c9cbf58882c41c4a0fea993014e27de7490b49e8b7",
"bytes": 2614907051,
},
"sft_data.jsonl": {
"lfs_sha256": "c1993d38c3a22a8bdfee65affc82d6559e5bb62e785b0f21c9151c75116151fc",
"bytes": 173137988,
},
"sft_images.zip": {
"lfs_sha256": "89ee34facc6793c51613613e0b10cac078942282f5fdec48d85751c6224bc3c2",
"bytes": 1026332147,
},
}
CLIP_FILE = {
"path": "pytorch_model.bin",
"lfs_sha256": "ec89c7b09c749a60aae3c9cd910516f24b58214a7df060b48962d14c469cfbf0",
"bytes": 598641023,
}
IMAGE_FILES = {
"Rainbow-Falls.jpg": "彩虹瀑布-Rainbow-Falls.jpg",
"Dog-Woman-Sea.jpg": "小狗美女海边-Dog-Woman-Sea.jpg",
"dance.jpg": "舞蹈-dance.jpg",
"Astronaut-Space.jpg": "太空宇航员-Astronaut-Space.jpg",
"city-traffic.jpg": "城市车水马龙-city-traffic.jpg",
"Panda-Grassland.jpg": "熊猫草地-Panda-Grassland.jpg",
"Bicycle-Flowers.jpg": "自行车鲜花-Bicycle-Flowers.jpg",
"Chair-Elderly-Reading.jpg": "椅子老人看书-Chair-Elderly-Reading.jpg",
}
IMAGE_SHA256 = {
"Rainbow-Falls.jpg": "1c8b74debaceb2e0bb6171b182084afe49288a0cc8089eb91eac69d067c27b10",
"Dog-Woman-Sea.jpg": "ba90d8b8738a44eac70811be5c89f767492b167ad4f6f6c31aa4591837d7e3dc",
"dance.jpg": "939e3132c8d3aec81f66f8aa928b476aaa25e00d94f1097f4974e73c913d5d8c",
"Astronaut-Space.jpg": "f466cdafecbdb85d2bad586896db5db3313afe18f9b3505667756cd25b747747",
"city-traffic.jpg": "73e90d82fbc5b1cf43b40de782b443f93f43a34e66b8ddebf3146d5dc1f83e00",
"Panda-Grassland.jpg": "0b7610a881039f0effdbfa46e9bb189132443d3ce2956856e8adf66d1ca22f8c",
"Bicycle-Flowers.jpg": "44fae0fafcd52c20b9bcaded897facbff00f61019cdd0aea543addf8499ad899",
"Chair-Elderly-Reading.jpg": "8fe91a90e837c33230d21cfe7ba5020e71b3ae99ac4c3fbd6d32cb54f51def53",
}
CONFIGS = (
"without_muon_pretrained",
"without_muon_sft",
"muon_from_dpo_pretrained",
"muon_from_dpo_sft",
"muon_from_pretrain_pretrained",
"muon_from_pretrain_sft",
"muon_from_sft_pretrained",
"muon_from_sft_sft",
)
CONFIG_META = {
"without_muon_pretrained": {
"architecture": "original",
"base_llm_stage": "sft",
"vlm_stage": "pretrained",
},
"without_muon_sft": {"architecture": "original", "base_llm_stage": "sft", "vlm_stage": "sft"},
"muon_from_dpo_pretrained": {
"architecture": "qk_norm_muon",
"base_llm_stage": "dpo",
"vlm_stage": "pretrained",
},
"muon_from_dpo_sft": {
"architecture": "qk_norm_muon",
"base_llm_stage": "dpo",
"vlm_stage": "sft",
},
"muon_from_pretrain_pretrained": {
"architecture": "qk_norm_muon",
"base_llm_stage": "pretrain",
"vlm_stage": "pretrained",
},
"muon_from_pretrain_sft": {
"architecture": "qk_norm_muon",
"base_llm_stage": "pretrain",
"vlm_stage": "sft",
},
"muon_from_sft_pretrained": {
"architecture": "qk_norm_muon",
"base_llm_stage": "sft",
"vlm_stage": "pretrained",
},
"muon_from_sft_sft": {
"architecture": "qk_norm_muon",
"base_llm_stage": "sft",
"vlm_stage": "sft",
},
}
SECTION_SPECS = (
(
"## Without Muon Optimizer",
(
("without_muon_pretrained", "### Pretrained VLM"),
("without_muon_sft", "### VLM after SFT"),
),
),
(
"## VLM with Muon Optimizer (from DPO)",
(
("muon_from_dpo_pretrained", "### Pretrained VLM"),
("muon_from_dpo_sft", "### VLM with SFT"),
),
),
(
"## VLM with Muon Optimizer (from Pretrain)",
(
("muon_from_pretrain_pretrained", "### Pretrained VLM"),
("muon_from_pretrain_sft", "### VLM with SFT"),
),
),
(
"## VLM with Muon Optimizer (from SFT)",
(
("muon_from_sft_pretrained", "### Pretrained VLM"),
("muon_from_sft_sft", "### VLM with SFT"),
),
),
)
LABELS = tuple("ABCDEFGH")
METRICS = ("grounding_accuracy", "hallucination_control", "coverage", "visual_specificity")
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def canonical_json_bytes(value: Any) -> bytes:
return (json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode()
def write_json(path: Path, value: Any) -> None:
path.write_bytes(canonical_json_bytes(value))
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def _parse_code_block(block: str, config: str) -> dict[str, Any]:
fence = re.search(r"```[^\n]*\n(.*?)\n```", block, flags=re.DOTALL)
if not fence:
raise ValueError(f"{config}: no evaluation code block")
transcript = fence.group(1).strip()
matches = list(
re.finditer(
r"(?m)^\[Image\]:\s*([^\n]+)\n🤖️:\s*(.*?)(?=\n(?:\s*\n)*\[Image\]:|\Z)",
transcript,
flags=re.DOTALL,
)
)
outputs = []
for match in matches:
image = match.group(1).strip()
output = match.group(2).strip()
if image not in IMAGE_FILES:
raise ValueError(f"{config}: unexpected image {image!r}")
if not output:
raise ValueError(f"{config}/{image}: empty output")
outputs.append({"image": image, "output": output})
if len(outputs) != len(IMAGE_FILES) or {row["image"] for row in outputs} != set(IMAGE_FILES):
raise ValueError(f"{config}: expected all eight images, found {len(outputs)}")
command = next(
(line.strip() for line in transcript.splitlines() if line.strip().startswith("$")), ""
)
return {
"config": config,
**CONFIG_META[config],
"historical_command": command,
"output_count": len(outputs),
"outputs": outputs,
}
def parse_retained_outputs(report_path: Path = REPORT_PATH) -> dict[str, Any]:
text = report_path.read_text(encoding="utf-8")
start = text.index("# Analysis of Vision-Language Model Training Results")
end = text.index("## Key Findings and Summary of VLM Training", start)
vlm = text[start:end]
cells = []
for section_index, (section_heading, stages) in enumerate(SECTION_SPECS):
section_start = vlm.index(section_heading)
section_end = (
vlm.index(SECTION_SPECS[section_index + 1][0], section_start)
if section_index + 1 < len(SECTION_SPECS)
else len(vlm)
)
section = vlm[section_start:section_end]
for stage_index, (config, heading) in enumerate(stages):
cell_start = section.index(heading)
cell_end = (
section.index(stages[stage_index + 1][1], cell_start)
if stage_index + 1 < len(stages)
else len(section)
)
cells.append(_parse_code_block(section[cell_start:cell_end], config))
if tuple(cell["config"] for cell in cells) != CONFIGS:
raise ValueError("historical VLM cells are incomplete or out of order")
return {
"schema_version": "exp8-4-retained-outputs-v1",
"experiment": "8-4",
"source_report": str(report_path.relative_to(REPO_ROOT)),
"source_report_sha256": sha256_file(report_path),
"cell_count": len(cells),
"output_count": sum(cell["output_count"] for cell in cells),
"images": list(IMAGE_FILES),
"configs": list(CONFIGS),
"cells": cells,
}
def outputs_for_image(retained: dict[str, Any], image: str) -> dict[str, str]:
rows = {}
for cell in retained["cells"]:
match = [row for row in cell["outputs"] if row["image"] == image]
if len(match) != 1:
raise ValueError(f"{cell['config']}/{image}: expected one retained output")
rows[cell["config"]] = match[0]["output"]
return rows
def blind_mapping(image: str) -> dict[str, str]:
configs = list(CONFIGS)
image_seed = int(hashlib.sha256(image.encode()).hexdigest()[:8], 16)
random.Random(BLIND_SEED + image_seed).shuffle(configs)
return dict(zip(LABELS, configs, strict=True))
def image_path(source_dir: Path, image: str) -> Path:
return source_dir / "dataset" / "eval_images" / IMAGE_FILES[image]
def image_data_url(path: Path) -> str:
mime = mimetypes.guess_type(path.name)[0] or "image/jpeg"
return f"data:{mime};base64,{base64.b64encode(path.read_bytes()).decode()}"
def judge_payload(
retained: dict[str, Any], image: str, source_dir: Path, model: str
) -> tuple[dict[str, Any], dict[str, str]]:
mapping = blind_mapping(image)
outputs = outputs_for_image(retained, image)
candidates = {label: outputs[config] for label, config in mapping.items()}
required = {
"image": image,
"candidates": {
label: {
**{metric: "number 0-5" for metric in METRICS},
"material_errors": ["specific image-grounding errors; empty only if none"],
"rationale": "brief evidence-based explanation",
}
for label in LABELS
},
"rank_order": list(LABELS),
"best": "one label A-H",
}
text = json.dumps(
{
"image": image,
"task": "Judge eight anonymous captions against the attached image.",
"rubric": {
"grounding_accuracy": "0 unrelated or false; 3 main subject mostly right; 5 all material claims visibly supported",
"hallucination_control": "0 dominated by invented objects/relations; 3 some speculation; 5 no material invention",
"coverage": "0 misses the scene; 3 covers main subject; 5 covers the important visible scene without padding",
"visual_specificity": "0 generic/nonvisual; 3 some concrete details; 5 precise discriminative visible details",
},
"candidates": candidates,
"required_json_shape": required,
},
ensure_ascii=False,
sort_keys=True,
)
payload = {
"model": model,
"temperature": 0,
"response_format": {"type": "json_object"},
"messages": [
{
"role": "system",
"content": (
"You are an independent vision-language evaluator. The candidates are anonymous. "
"Do not infer model identity, optimizer, base checkpoint, or training stage. Inspect "
"the attached image, score only visible grounding, and return exactly one JSON object."
),
},
{
"role": "user",
"content": [
{
"type": "image_url",
"image_url": {
"url": image_data_url(image_path(source_dir, image)),
"detail": "high",
},
},
{"type": "text", "text": text},
],
},
],
}
return payload, mapping
def extract_json_object(content: str) -> dict[str, Any]:
stripped = content.strip()
if stripped.startswith("```"):
stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
stripped = re.sub(r"\s*```$", "", stripped)
try:
parsed = json.loads(stripped)
except json.JSONDecodeError:
start = stripped.find("{")
if start < 0:
raise
parsed, _ = json.JSONDecoder().raw_decode(stripped[start:])
if not isinstance(parsed, dict):
raise TypeError("judge content must decode to an object")
return parsed
def validate_judgment(judgment: dict[str, Any], image: str) -> None:
if judgment.get("image") != image:
raise ValueError(f"judge returned wrong image: {judgment.get('image')!r}")
candidates = judgment.get("candidates")
if not isinstance(candidates, dict) or set(candidates) != set(LABELS):
raise ValueError("judge must score A-H exactly")
for label in LABELS:
row = candidates[label]
if not isinstance(row, dict):
raise TypeError(f"candidate {label} score must be an object")
for metric in METRICS:
value = row.get(metric)
if (
not isinstance(value, (int, float))
or isinstance(value, bool)
or not 0 <= value <= 5
):
raise ValueError(f"candidate {label} invalid {metric}: {value!r}")
if not isinstance(row.get("material_errors"), list):
raise TypeError(f"candidate {label} material_errors must be a list")
if not isinstance(row.get("rationale"), str) or not row["rationale"].strip():
raise ValueError(f"candidate {label} rationale is missing")
rank_order = judgment.get("rank_order")
if (
not isinstance(rank_order, list)
or len(rank_order) != len(LABELS)
or set(rank_order) != set(LABELS)
):
raise ValueError("rank_order must be a permutation of A-H")
if judgment.get("best") not in LABELS:
raise ValueError("best must be one label A-H")
def call_judge(
retained: dict[str, Any],
image: str,
*,
source_dir: Path,
endpoint: str,
model: str,
api_key: str,
timeout: float,
) -> dict[str, Any]:
payload, mapping = judge_payload(retained, image, source_dir, model)
request = urllib.request.Request(
endpoint,
data=json.dumps(payload, ensure_ascii=False).encode(),
method="POST",
headers={"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"},
)
started = time.perf_counter()
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
raw_body = response.read()
http_status = response.status
except urllib.error.HTTPError as exc:
body = exc.read().decode(errors="replace")
raise RuntimeError(f"judge HTTP {exc.code}: {body[:500]}") from exc
latency_ms = round((time.perf_counter() - started) * 1000, 3)
raw_response = json.loads(raw_body)
judgment = extract_json_object(raw_response["choices"][0]["message"]["content"])
validate_judgment(judgment, image)
response_id = raw_response.get("id")
usage = raw_response.get("usage")
if not isinstance(response_id, str) or not response_id:
raise ValueError("judge response has no response ID")
if not isinstance(usage, dict) or not isinstance(usage.get("total_tokens"), int):
raise TypeError("judge response has no complete usage")
path = image_path(source_dir, image)
return {
"image": image,
"image_source_filename": IMAGE_FILES[image],
"image_sha256": sha256_file(path),
"image_bytes": path.stat().st_size,
"provider": "ark",
"endpoint": endpoint,
"credential_env": "ARK_API_KEY",
"credential_headers_retained": False,
"blind_seed": BLIND_SEED,
"blind_map": mapping,
"request": payload,
"http_status": http_status,
"response": raw_response,
"response_id": response_id,
"usage": usage,
"latency_ms": latency_ms,
"judgment": judgment,
}
def reproduction_contract() -> dict[str, Any]:
return {
"schema_version": "exp8-4-reproduction-contract-v1",
"experiment": "8-4",
"historical_evidence_boundary": {
"historical_training_executed": True,
"eight_historical_cells_and_64_outputs_retained": True,
"historical_source_revisions_retained": False,
"historical_dataset_hashes_retained": False,
"historical_base_checkpoint_hashes_retained": False,
"historical_vlm_checkpoint_hashes_retained": False,
"historical_rng_and_stepwise_logs_retained": False,
"claim": (
"The author's report establishes that eight VLM configurations were evaluated on eight images. "
"It does not establish byte identity of the historical code, datasets, base/VLM checkpoints, or RNG state."
),
},
"future_reproduction": {
"vlm_source": {
"repository": "bojieli/minimind-v",
"original_revision": ORIGINAL_VLM_REVISION,
"original_files_sha256": ORIGINAL_VLM_FILES,
"qk_norm_muon_revision": IMPROVED_VLM_REVISION,
"qk_norm_muon_files_sha256": IMPROVED_VLM_FILES,
"not_claimed_as_historical_revisions": True,
},
"base_llm_source": {
"repository": "bojieli/minimind",
"original_revision": ORIGINAL_LLM_REVISION,
"original_files_sha256": ORIGINAL_LLM_FILES,
"qk_norm_muon_revision": IMPROVED_LLM_REVISION,
"qk_norm_muon_files_sha256": IMPROVED_LLM_FILES,
"dependency": "Use the Experiment 8-3 data/commands to produce original-SFT and improved pretrain/SFT/DPO 768-dimension base checkpoints.",
},
"vlm_dataset": {
"repository": "jingyaogong/minimind-v_dataset",
"revision": DATASET_REVISION,
"selected_for_jsonl_script_compatibility": True,
"files": DATASET_FILES,
},
"vision_encoder": {
"repository": "openai/clip-vit-base-patch16",
"revision": CLIP_REVISION,
"file": CLIP_FILE,
},
"evaluation_images": {
image: {"source_filename": IMAGE_FILES[image], "sha256": IMAGE_SHA256[image]}
for image in IMAGE_FILES
},
"commands": {
"original_source": "git clone https://github.com/bojieli/minimind-v.git sources/original-minimind-v && git -C sources/original-minimind-v checkout --detach 765908051d0837d60cecfb93f8390334e2e55f1e",
"improved_source": "git clone https://github.com/bojieli/minimind-v.git sources/qk-norm-muon-minimind-v && git -C sources/qk-norm-muon-minimind-v checkout --detach ead791c530fa5f9a3549dbfe9e11ec732d18d2e5",
"dataset": "git clone https://huggingface.co/datasets/jingyaogong/minimind-v_dataset dataset-source && git -C dataset-source checkout --detach ac9d03a3fd26a2d8e74bda374d9a2ddba49e4c1b && cp dataset-source/{pretrain_data.jsonl,sft_data.jsonl} dataset/ && unzip dataset-source/pretrain_images.zip -d dataset && unzip dataset-source/sft_images.zip -d dataset",
"vision_encoder": "git clone https://huggingface.co/openai/clip-vit-base-patch16 model/vision_model/clip-vit-base-patch16 && git -C model/vision_model/clip-vit-base-patch16 checkout --detach 57c216476eefef5ab752ec549e440a49ae4ae5f3",
"original_pretrain_vlm": "install -m 0644 <exp8-3-original-sft-768.pth> runs/original/out/llm_768.pth && cd trainer && torchrun --nproc_per_node=8 train_pretrain_vlm.py --out_dir ../runs/original/out --epochs 4 --hidden_size 768 --num_hidden_layers 16 --data_path ../dataset/pretrain_data.jsonl --images_path ../dataset/pretrain_images --use_wandb",
"original_sft_vlm": "cd trainer && torchrun --nproc_per_node=8 train_sft_vlm.py --out_dir ../runs/original/out --epochs 4 --hidden_size 768 --num_hidden_layers 16 --data_path ../dataset/sft_data.jsonl --images_path ../dataset/sft_images --use_wandb",
"improved_matrix": "For each BASE in pretrain,sft,dpo, install the corresponding Experiment-8-3 QK-Norm+Muon 768-dimension checkpoint as runs/muon-from-$BASE/out/llm_768.pth, then run train_pretrain_vlm_muon.py and train_sft_vlm_muon.py with the same four-epoch data arguments in that isolated out_dir.",
"evaluation": "For every isolated out_dir, preserve both checkpoints, copy the selected *_muon_768.pth name to eval_vlm.py's pretrain_vlm_768.pth or sft_vlm_768.pth compatibility name when needed, then run python eval_vlm.py --load 0 --model_mode 0 and --model_mode 1 on the eight hash-pinned images with seed 1337.",
},
"environment": {
"book_pyproject": "pyproject.toml",
"book_lock": "uv.lock",
"install": "uv sync --locked --python 3.12 --extra ch7 --extra dev",
"boundary": "The book lock freezes a future Python environment; CUDA, drivers, and the historical GPU image were not retained.",
},
},
"reported_training_design": {
"parameter_count_millions": {"original": 104.622, "qk_norm_muon": 104.625},
"base_llm_stages": ["pretrain", "sft", "dpo"],
"vlm_stages": ["pretrained", "sft"],
"projection_pretraining_freezes_llm": True,
"sft_unfreezes_full_model": True,
"reported_epochs": 4,
"seed_in_current_source": 1337,
"source_verified_mechanisms": {
"original_revision_precedes_qk_norm_commit": True,
"improved_revision_has_qk_norm_before_rope": True,
"improved_revision_uses_muon_for_selected_2d_weights": True,
"vision_encoder_is_frozen_clip": True,
},
},
"checkpoint_policy": {
"distributed_with_book": False,
"acceptance_artifact": False,
"required_artifact": "reproducible evidence-backed training report",
"reason": "Training checkpoints are intentionally not distributed to readers.",
},
}
def mean(values: list[float]) -> float:
return round(sum(values) / len(values), 4)
def summarize(
retained: dict[str, Any], receipts: list[dict[str, Any]], contract: dict[str, Any]
) -> dict[str, Any]:
per_image_config_scores: dict[str, dict[str, Any]] = {}
best_counts = {config: 0 for config in CONFIGS}
for receipt in receipts:
scores = {
receipt["blind_map"][label]: row
for label, row in receipt["judgment"]["candidates"].items()
}
per_image_config_scores[receipt["image"]] = scores
best_counts[receipt["blind_map"][receipt["judgment"]["best"]]] += 1
config_averages = {}
for config in CONFIGS:
rows = [per_image_config_scores[image][config] for image in IMAGE_FILES]
config_averages[config] = {
metric: mean([float(row[metric]) for row in rows]) for metric in METRICS
}
config_averages[config]["overall"] = mean(
[float(row[metric]) for row in rows for metric in METRICS]
)
stage_averages = {}
for stage in ("pretrained", "sft"):
configs = [config for config in CONFIGS if CONFIG_META[config]["vlm_stage"] == stage]
stage_averages[stage] = {
metric: mean([config_averages[config][metric] for config in configs])
for metric in (*METRICS, "overall")
}
isolated_pairs = {}
for stage in ("pretrained", "sft"):
original = f"without_muon_{stage}"
improved = f"muon_from_sft_{stage}"
isolated_pairs[stage] = {
"original": config_averages[original]["overall"],
"qk_norm_muon_from_sft": config_averages[improved]["overall"],
"delta": round(
config_averages[improved]["overall"] - config_averages[original]["overall"], 4
),
}
response_ids = [receipt["response_id"] for receipt in receipts]
acceptance = {
"historical_report_content_hashed": bool(retained["source_report_sha256"]),
"all_eight_configuration_cells_retained": retained["cell_count"] == 8,
"all_64_historical_outputs_retained": retained["output_count"] == 64,
"same_eight_images_present_in_every_cell": all(
cell["output_count"] == 8 for cell in retained["cells"]
),
"eight_image_aware_arm_blind_judgments": len(receipts) == 8,
"raw_judge_requests_responses_ids_usage_latency_retained": len(set(response_ids)) == 8
and all(
receipt["usage"].get("total_tokens", 0) > 0 and receipt["latency_ms"] > 0
for receipt in receipts
),
"request_images_match_pinned_sha256": all(
receipt["image_sha256"] == IMAGE_SHA256[receipt["image"]] for receipt in receipts
),
"immutable_original_and_improved_source_revisions_frozen": bool(
ORIGINAL_VLM_REVISION and IMPROVED_VLM_REVISION
),
"immutable_dataset_clip_and_eval_image_inputs_frozen": bool(
DATASET_REVISION and CLIP_REVISION and IMAGE_SHA256
),
"future_reproduction_commands_declared": len(contract["future_reproduction"]["commands"])
>= 6,
"historical_provenance_limitations_explicit": contract["historical_evidence_boundary"][
"historical_vlm_checkpoint_hashes_retained"
]
is False,
"checkpoints_not_an_acceptance_artifact": contract["checkpoint_policy"][
"acceptance_artifact"
]
is False,
}
passed = all(acceptance.values())
ranking = sorted(CONFIGS, key=lambda config: (-config_averages[config]["overall"], config))
return {
"schema_version": "exp8-4-summary-v1",
"experiment": "8-4",
"status": "passed" if passed else "failed",
"judge": {
"provider": "ark",
"model": receipts[0]["request"]["model"],
"image_aware": True,
"calls": len(receipts),
"response_ids": response_ids,
"total_tokens": sum(receipt["usage"]["total_tokens"] for receipt in receipts),
"total_latency_ms": round(sum(receipt["latency_ms"] for receipt in receipts), 3),
"blind_seed": BLIND_SEED,
},
"retained": {
"cells": retained["cell_count"],
"outputs": retained["output_count"],
"images": len(retained["images"]),
},
"config_averages": config_averages,
"stage_averages": stage_averages,
"isolated_original_vs_qk_norm_muon_from_sft": isolated_pairs,
"ranking_by_overall": ranking,
"best_counts": best_counts,
"per_image_config_scores": per_image_config_scores,
"scientific_findings": {
"top_configuration": ranking[0],
"top_configuration_overall": config_averages[ranking[0]]["overall"],
"sft_minus_pretrained_average": round(
stage_averages["sft"]["overall"] - stage_averages["pretrained"]["overall"], 4
),
"author_claims_are_historical_observations_not_acceptance_gates": True,
"muon_only_causal_claim_avoided": True,
},
"acceptance": {**acceptance, "passed": passed},
"limitations": [
"Historical base-LLM and VLM checkpoints are intentionally not distributed and were not recreated in this audit.",
"Historical source revisions, dataset identities, RNG state, hardware image, and stepwise logs were not retained.",
"Current immutable pins define a future reproduction contract and are not represented as the exact historical run.",
"The English captions are translations in a bilingual report, so translation can affect judging.",
"One image-aware judge call evaluates all eight anonymous candidates per image; scores are descriptive, not a powered significance test.",
"QK-Norm and Muon change together in the improved arm, so the report does not attribute effects to Muon alone.",
],
}
def render_report(summary: dict[str, Any]) -> str:
lines = [
"# Experiment 8-4 retained-training-report audit",
"",
"## Result",
"",
f"Status: **{summary['status']}**. The historical report retains {summary['retained']['outputs']} image descriptions across {summary['retained']['cells']} configurations and the same {summary['retained']['images']} images. Each image was inspected by a real image-capable ARK judge together with all eight arm-blind captions.",
"",
"| Configuration | Grounding | Hallucination control | Coverage | Specificity | Overall | Best count |",
"| --- | ---: | ---: | ---: | ---: | ---: | ---: |",
]
for config in summary["ranking_by_overall"]:
row = summary["config_averages"][config]
lines.append(
f"| {config} | {row['grounding_accuracy']:.4f} | {row['hallucination_control']:.4f} | {row['coverage']:.4f} | {row['visual_specificity']:.4f} | {row['overall']:.4f} | {summary['best_counts'][config]} |"
)
findings = summary["scientific_findings"]
lines.extend(
[
"",
f"The highest descriptive judge mean was **{findings['top_configuration']}** at **{findings['top_configuration_overall']:.4f}**. Averaged across all four base configurations, full VLM SFT changed the score by **{findings['sft_minus_pretrained_average']:+.4f}** versus projection-only pretraining.",
"",
"The isolated report comparison pairs original/SFT-base against QK-Norm+Muon/SFT-base at each VLM stage. QK-Norm and Muon still change together, so no Muon-only causal claim is made. All author-written qualitative claims remain historical observations rather than pass/fail gates.",
"",
"## Provenance and reproduction boundary",
"",
"`reproduction_contract.json` freezes separate pre-QK-Norm and QK-Norm+Muon MiniMind-V revisions, the corresponding base-LLM revisions, script-compatible VLM dataset Git-LFS objects, the CLIP weight object, all eight evaluation-image hashes, and future commands. These pins are not misrepresented as the historical checkout.",
"",
"Training checkpoints remain local by book policy and are not acceptance artifacts. The accepted artifact is this content-hashed report, all 64 retained outputs, eight raw image-aware judge receipts, and explicit limitations.",
"",
]
)
return "\n".join(lines)
def input_record(path: Path) -> dict[str, Any]:
return {
"path": str(path.relative_to(REPO_ROOT)),
"sha256": sha256_file(path),
"bytes": path.stat().st_size,
}
def artifact_record(path: Path, run_dir: Path) -> dict[str, Any]:
return {
"path": str(path.relative_to(run_dir)),
"sha256": sha256_file(path),
"bytes": path.stat().st_size,
}
def build_manifest(run_id: str, run_dir: Path, summary: dict[str, Any]) -> dict[str, Any]:
inputs = [
input_record(REPORT_PATH),
input_record(REPO_ROOT / "pyproject.toml"),
input_record(REPO_ROOT / "uv.lock"),
input_record(HERE / "run_vlm_training_report_audit.py"),
input_record(HERE / "validate_vlm_evidence.py"),
input_record(HERE / "test_vlm_training_report_audit.py"),
]
artifacts = [
run_dir / name
for name in (
"retained_outputs.json",
"reproduction_contract.json",
"judge_receipts.json",
"summary.json",
"report.md",
)
]
return {
"schema_version": "exp8-4-manifest-v1",
"experiment": "8-4",
"run_id": run_id,
"created_at": utc_now(),
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"inputs": inputs,
"artifacts": [artifact_record(path, run_dir) for path in artifacts],
"acceptance": summary["acceptance"],
"checkpoint_policy": "not distributed; not an acceptance artifact",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--run-id", default=DEFAULT_RUN_ID)
parser.add_argument("--source-dir", type=Path, default=os.getenv("MINIMIND_V_SOURCE_DIR"))
parser.add_argument("--endpoint", default=os.getenv("ARK_BASE_URL", DEFAULT_ENDPOINT))
parser.add_argument("--model", default=os.getenv("ARK_VISION_MODEL", DEFAULT_MODEL))
parser.add_argument("--api-key-env", default="ARK_API_KEY")
parser.add_argument("--timeout", type=float, default=240.0)
parser.add_argument("--concurrency", type=int, default=4)
parser.add_argument(
"--refresh-manifest",
action="store_true",
help="Rehash an existing run without provider calls; refuses changed retained outputs.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
if not re.fullmatch(r"[A-Za-z0-9._-]+", args.run_id):
raise SystemExit("run ID may contain only letters, digits, dot, underscore, and hyphen")
run_dir = RUNS_DIR / args.run_id
if args.refresh_manifest:
if not run_dir.is_dir():
raise SystemExit(f"cannot refresh missing run: {run_dir}")
stored = json.loads((run_dir / "retained_outputs.json").read_text(encoding="utf-8"))
retained = parse_retained_outputs()
if stored.get("cells") != retained.get("cells"):
raise SystemExit(
"refusing manifest refresh because retained historical outputs changed"
)
receipts_doc = json.loads((run_dir / "judge_receipts.json").read_text(encoding="utf-8"))
receipts = receipts_doc.get("calls")
if (
receipts_doc.get("schema_version") != "exp8-4-judge-receipts-v1"
or receipts_doc.get("experiment") != "8-4"
or not isinstance(receipts, list)
):
raise SystemExit("cannot refresh malformed judge receipts")
contract = reproduction_contract()
summary = summarize(retained, receipts, contract)
write_json(run_dir / "retained_outputs.json", retained)
write_json(run_dir / "reproduction_contract.json", contract)
write_json(run_dir / "summary.json", summary)
(run_dir / "report.md").write_text(render_report(summary), encoding="utf-8")
write_json(run_dir / "manifest.json", build_manifest(args.run_id, run_dir, summary))
latest = {
"schema_version": "exp8-4-latest-v1",
"experiment": "8-4",
"run_id": args.run_id,
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"manifest_sha256": sha256_file(run_dir / "manifest.json"),
}
write_json(LATEST_PATH, latest)
print(json.dumps(latest, indent=2, sort_keys=True))
return 0
if run_dir.exists():
raise SystemExit(f"refusing to overwrite existing run: {run_dir}")
if args.source_dir is None:
raise SystemExit("--source-dir or MINIMIND_V_SOURCE_DIR is required")
source_dir = args.source_dir.resolve()
for image, expected in IMAGE_SHA256.items():
path = image_path(source_dir, image)
if not path.is_file() or sha256_file(path) != expected:
raise SystemExit(f"evaluation image missing or hash mismatch: {path}")
api_key = os.getenv(args.api_key_env)
if not api_key:
raise SystemExit(f"missing required credential environment variable: {args.api_key_env}")
retained = parse_retained_outputs()
with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as pool:
futures = {
image: pool.submit(
call_judge,
retained,
image,
source_dir=source_dir,
endpoint=args.endpoint,
model=args.model,
api_key=api_key,
timeout=args.timeout,
)
for image in IMAGE_FILES
}
receipts = [futures[image].result() for image in IMAGE_FILES]
contract = reproduction_contract()
summary = summarize(retained, receipts, contract)
run_dir.mkdir(parents=True)
write_json(run_dir / "retained_outputs.json", retained)
write_json(
run_dir / "judge_receipts.json",
{
"schema_version": "exp8-4-judge-receipts-v1",
"experiment": "8-4",
"credential_headers_retained": False,
"calls": receipts,
},
)
write_json(run_dir / "reproduction_contract.json", contract)
write_json(run_dir / "summary.json", summary)
(run_dir / "report.md").write_text(render_report(summary), encoding="utf-8")
write_json(run_dir / "manifest.json", build_manifest(args.run_id, run_dir, summary))
latest = {
"schema_version": "exp8-4-latest-v1",
"experiment": "8-4",
"run_id": args.run_id,
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"manifest_sha256": sha256_file(run_dir / "manifest.json"),
}
write_json(LATEST_PATH, latest)
print(json.dumps(latest, indent=2, sort_keys=True))
return 0 if summary["status"] == "passed" else 1
if __name__ == "__main__":
raise SystemExit(main())
chapter8/MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/judge_receipts.json
+804
View File
File diff suppressed because one or more lines are too long
chapter8/MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/manifest.json
+79
View File
@@ -0,0 +1,79 @@
{
"acceptance": {
"all_expected_outputs_retained": true,
"all_six_arm_stage_cells_retained": true,
"checkpoints_not_an_acceptance_artifact": true,
"eight_stage_balanced_blind_judgments": true,
"future_reproduction_commands_declared": true,
"historical_provenance_limitations_explicit": true,
"immutable_dataset_revision_lfs_hashes_and_sizes_frozen": true,
"immutable_source_revision_and_file_hashes_frozen": true,
"original_and_qk_norm_muon_compared": true,
"passed": true,
"pretrain_sft_and_dpo_compared": true,
"raw_historical_report_hashed": true,
"raw_judge_requests_responses_ids_usage_latency_retained": true,
"reported_loss_claims_qualified": true
},
"artifacts": [
{
"bytes": 41195,
"path": "retained_outputs.json",
"sha256": "c38ba7fcf766e15e43712ceb7d99ccadcbea4400caf18d28993d7a3f7c6dc88a"
},
{
"bytes": 5555,
"path": "reproduction_contract.json",
"sha256": "224df3576cf8751ef1a1ceb6d879db0afec1347a79c094a02c372982aa3068e3"
},
{
"bytes": 90942,
"path": "judge_receipts.json",
"sha256": "e254be18e34aab27812d529c330490956545be3b66b5b59eb0958fb3c45e8e5b"
},
{
"bytes": 12556,
"path": "summary.json",
"sha256": "d06d354c8ed8e2d84c506b96dd7bdea0a6cfff36be24b2d731d5168855228740"
},
{
"bytes": 1464,
"path": "report.md",
"sha256": "a62514227ffb7770eb84dbe5f965df7f215582b8b481d13a7beebf9a7a4b28dd"
}
],
"checkpoint_policy": "not distributed; not an acceptance artifact",
"created_at": "2026-08-17T05:37:04.936375+00:00",
"experiment": "8-3",
"inputs": [
{
"bytes": 155255,
"path": "chapter8/MiniMind-pretrain/README.md",
"sha256": "a8e1df1a9ee5cf013995e9ff3b963621485a838c97456b9f30e65ea9fdf55d50"
},
{
"bytes": 14150,
"path": "pyproject.toml",
"sha256": "33cc27c2759f353663d6907f1a918a41a49fb8d3e4e28370d508136449c3b156"
},
{
"bytes": 1311181,
"path": "uv.lock",
"sha256": "347fc87f40526372c284c61e5374536ce8d2071936cef8844d350470ecdf0d0b"
},
{
"bytes": 34784,
"path": "chapter8/MiniMind-pretrain/validation/run_training_report_audit.py",
"sha256": "b2d43301a633cac2fa1fb7d29bfa831837b8dc7b27b2f887a0b70e45d68570f1"
},
{
"bytes": 14855,
"path": "chapter8/MiniMind-pretrain/validation/validate_evidence.py",
"sha256": "cc49d5185fae5680a5aac4d27b831b8a7215f3d7339ac46a998812e81bc13c9b"
}
],
"run_dir": "validation/runs/exp8-3-training-report-20260731-v1",
"run_id": "exp8-3-training-report-20260731-v1",
"schema_version": "exp8-3-manifest-v1",
"status": "passed"
}
chapter8/MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/report.md
+20
View File
@@ -0,0 +1,20 @@
# Experiment 8-3 retained-training-report audit
## Result
Status: **passed**. The historical report retains 49 outputs across the original and QK-Norm + Muon arms after pretrain, SFT, and DPO. Eight preregistered arm-blind comparisons were judged from raw retained text by an independent ARK model.
| Arm | Fluency | Instruction | Factuality | Overall |
| --- | ---: | ---: | ---: | ---: |
| Original | 3.0000 | 1.7500 | 1.3750 | 2.0417 |
| QK-Norm + Muon | 3.7500 | 3.0000 | 4.1250 | 3.6250 |
Observed blind-judge overall delta: **+1.5833**. Pairwise decisions: {'original': 0, 'qk_norm_muon': 7, 'tie': 1}.
The report's loss claims (3.0 reached at 36 versus 12 reported steps; final loss 2.0 versus 1.7) are retained as author-reported observations, not independently recomputed measurements, because the historical stepwise logs were not preserved.
## Provenance and reproduction boundary
`reproduction_contract.json` freezes the MiniMind source revision, hashes the relevant source files, freezes a dataset revision with the three Git-LFS object hashes and sizes, and records all six future reproduction commands. These pins were selected for future reproduction and are not represented as the exact historical checkout.
Training checkpoints remain local by book policy and are not an acceptance artifact. The accepted artifact is this content-hashed training report, its raw retained outputs, raw independent-judge receipts, and explicit limitations.
chapter8/MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/reproduction_contract.json
+106
View File
@@ -0,0 +1,106 @@
{
"checkpoint_policy": {
"acceptance_artifact": false,
"distributed_with_book": false,
"reason": "Training checkpoints are intentionally not distributed to readers.",
"required_artifact": "reproducible evidence-backed training report"
},
"experiment": "8-3",
"future_reproduction": {
"commands": {
"improved_dpo": "torchrun --nproc_per_node=8 trainer/train_dpo.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --init_from out/full_sft_muon_768.pth --data_path dataset/dpo.jsonl --use_wandb",
"improved_pretrain": "torchrun --nproc_per_node=8 trainer/train_pretrain_muon.py --epochs 10 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/pretrain_hq.jsonl --use_wandb",
"improved_sft": "torchrun --nproc_per_node=8 trainer/train_full_sft_muon.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/sft_512.jsonl --use_wandb",
"original_dpo": "torchrun --nproc_per_node=8 trainer/train_dpo.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --init_from out/full_sft_768.pth --data_path dataset/dpo.jsonl --use_wandb",
"original_pretrain": "torchrun --nproc_per_node=8 trainer/train_pretrain.py --epochs 10 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/pretrain_hq.jsonl --use_wandb",
"original_sft": "torchrun --nproc_per_node=8 trainer/train_full_sft.py --epochs 2 --hidden_size 768 --num_hidden_layers 16 --data_path dataset/sft_512.jsonl --use_wandb"
},
"dataset": {
"files": {
"dpo.jsonl": {
"bytes": 53653322,
"lfs_sha256": "ee934a8a455ccc99d1334d63e1254dd1d64f497fd067cfcbb71e3043f5b46768"
},
"pretrain_hq.jsonl": {
"bytes": 1669750047,
"lfs_sha256": "9801b0d2210c61c2e4bc130f6dc4b3c870698a88d04af8f103c23dd5f0ce2440"
},
"sft_512.jsonl": {
"bytes": 7531517862,
"lfs_sha256": "053b7d09574e48a86232e929211434ff9e5016c6ed13312e63687dd52edcbebf"
}
},
"not_claimed_as_historical_revision": true,
"repository": "jingyaogong/minimind_dataset",
"revision": "84983ed4dec7836d240577760c1d6be5d4cabcf9",
"selected_at": "2026-07-31"
},
"environment": {
"book_lock": "uv.lock",
"book_pyproject": "pyproject.toml",
"boundary": "The book lock freezes a future software environment. The pinned upstream requirements file itself is unversioned, and the future GPU/CUDA stack has not been exercised here.",
"install": "uv sync --locked --python 3.12 --extra ch7 --extra dev"
},
"source": {
"file_sha256": {
"dataset/lm_dataset.py": "213726b1781289548784220b2b2db48fe97d84f3b67ac0cd70186cbbcb7b5d2c",
"eval_model.py": "43930a4b55048a4a3ffa17eb78ae67d59582d639aa9365f0bbf41ba149128af8",
"model/model_minimind.py": "2d33988711c704be6a22c4c61489b23106a2340a7cb8b97ebe3e40f30819cbb0",
"model/tokenizer.json": "e489029175fb3f94b8211a120a72a2ee41a664db65b828d077c7bde989c845a9",
"model/tokenizer_config.json": "190cc4738bac3b6f6b563376019c581b320fdb0260a03b9d5ab806296c8c6bb8",
"requirements.txt": "23f4cea09281765eec7cf03e28231425638e8e42580f418781fed166b75af968",
"trainer/muon.py": "00c2c6a225edeb55433df0724c3c74f6ff98ac4b2cc73c4aafcff686824f6267",
"trainer/train_dpo.py": "97f2c31cc8bc21a777e2efcb5e2fa35a49e4e9e3698db120148f8a0b2f678449",
"trainer/train_full_sft.py": "a57422f1df80bf2867f31f3b4a646a92ac7f66729e98a3b32cd1ec4d6780cb8b",
"trainer/train_full_sft_muon.py": "acd0b7db5b1d8b25d3c3103f92d68a7d381f9322a1b33bbec34be3d005930bad",
"trainer/train_pretrain.py": "ddd122645a9f1043bc8dac69a81ac51d2df95df8745d25faed7963d38fedc328",
"trainer/train_pretrain_muon.py": "fc83d07754ec3a8c156b6b8bfc0fd4326edecb72efabc5e08ae4ff5e3a7029bc"
},
"not_claimed_as_historical_revision": true,
"repository": "bojieli/minimind",
"revision": "8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795",
"selected_at": "2026-07-31"
}
},
"historical_evidence_boundary": {
"claim": "The author's retained report supports that original and QK-Norm+Muon 104.03M models were evaluated after pretrain, SFT, and DPO. It does not establish byte identity of the historical checkpoints, datasets, source checkout, or every loss point.",
"historical_checkpoint_hashes_retained": false,
"historical_dataset_hashes_retained": false,
"historical_source_revision_retained": false,
"historical_stepwise_training_logs_retained": false,
"historical_training_executed": true,
"six_historical_evaluation_transcripts_retained": true
},
"model_and_training": {
"architecture": {
"hidden_size": 768,
"layers": 16,
"sequence_length": 512
},
"arms": [
"original",
"qk_norm_muon"
],
"reported_parameter_count_millions": 104.03,
"reported_scalars_without_stepwise_logs": {
"eight_rtx_4090_pretrain_hours": 6,
"eight_rtx_4090_sft_hours": 8,
"final_loss_original": 2.0,
"final_loss_qk_norm_muon": 1.7,
"steps_to_loss_3_original": 36,
"steps_to_loss_3_qk_norm_muon": 12
},
"source_verified_mechanisms": {
"adamw_for_embeddings_norms_and_lm_head": true,
"dpo_uses_adamw_from_arm_specific_sft_checkpoint": true,
"muon_for_two_dimensional_non_embedding_weights": true,
"qk_norm_before_rope": true
},
"stages": [
"pretrain",
"sft",
"dpo"
]
},
"schema_version": "exp8-3-reproduction-contract-v1"
}
chapter8/MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/retained_outputs.json
+263
View File
@@ -0,0 +1,263 @@
{
"arms": [
"original",
"qk_norm_muon"
],
"cell_count": 6,
"cells": [
{
"arm": "original",
"pair_count": 7,
"pairs": [
{
"output": "and the relationship of universal laws are the premise of the basic principles of Marxism. The basic principles of Marxism are the relationship of universal laws. Basic principles refer to those determined by universal laws, where universal laws are \"universal laws.\" Universal laws refer to those determined by universal laws, but they are not absolute laws; they are only a form of rules, a part of universal laws. The basic principles of Marxism include universal laws, that is, those determined by universal laws, rather than absolute laws.",
"prompt": "Basic principles of Marxism"
},
{
"output": "What are they? The main functions of the human brain include:\n1. Memory: The formation and retrieval of memory are through chemical and electrical signals between neurons in the brain, storing and retrieving information.\n2. Decision-making: Decision-making is the ultimate goal of the brain. It is formed by the connections and interactions between neurons and neurons.\n3. Senses: Perception, sensation, action, and feeling are all closely related to brain function. Senses are the sensory information in our brain, including vision, hearing, smell, taste, smell, and touch. Senses are the sensory information in our brain, including vision, hearing, smell, taste, smell, and touch.\nThese functions and structures together help us complete complex thinking and behavior.",
"prompt": "The main functions of the human brain"
},
{
"output": "to study the interaction between celestial bodies. It firmly places all objects on a flat surface, forming a torque, called universal gravitation. The law of universal gravitation is one of the three basic laws discovered by Newton. It is a scientific theory describing how matter in the universe interacts. The law of universal gravitation describes the interaction between objects, placing all objects firmly on a flat surface, forming a torque, called Newton's law. It is one of the three basic laws discovered by Newton, a scientific theory describing how matter in the universe interacts.",
"prompt": "The principle of universal gravitation is"
},
{
"output": "which one? The highest mountain in the world is Mount Everest, located in the Himalayas, with an altitude of 8,848 meters.\nMount Everest is one of the most challenging climbing targets on Earth, attracting climbers from all over the world every year. Climbing Mount Everest requires climbers to overcome various challenges such as gravity, altitude, and physical strength. At the same time, Mount Everest is also one of the world's natural wonders, known as the \"Roof of the World.\"\nMount Everest is located on the border between Nepal and China, in the Himalayas. In Nepali, its name means \"Goddess's Home,\" meaning \"Goddess's Home.\" In the Himalayas, there are many peaks, the highest of which is Mount Everest, with an altitude of 8,848 meters.",
"prompt": "The highest mountain in the world is"
},
{
"output": "has a concentration of zero, right? Wrong.\nCarbon dioxide is a gas with the molecular formula CO2. It is a molecule composed of one carbon atom and two oxygen atoms. Carbon dioxide is a very important element, widely present in nature. It is also one of the main gases for respiration and digestion in humans and other animals.\nAlthough the concentration of carbon dioxide in the air is high, this does not mean it is a harmful chemical. It only accounts for about 20% of the air. Other carbon compounds (such as glucose and starch) can also have a negative impact on human health.\nTherefore, carbon dioxide is not a harmful chemical.",
"prompt": "Carbon dioxide in the air"
},
{
"output": "7, what are their sizes? This number is relatively common, because there are many types of animals on Earth, but the answer is 7.\nIn fact, the largest animal on Earth is the blue whale, which can reach a size of over 100 feet (30 meters). The blue whale is a marine mammal, often considered a sea giant because they live in the ocean. The blue whale is currently the largest known animal, with a weight of up to 200 tons and a length of up to 30 meters.\nIn addition, there are many other different types of organisms in the ocean, such as plankton, mollusks, etc. The sizes and shapes of these organisms vary, some can even grow to over 300 feet, and some can even reach 40 feet.\nIn summary, although we cannot directly classify all animals on Earth into one category, we can be sure that there are many different types of animals on Earth.",
"prompt": "The largest animal on Earth has"
},
{
"output": "what? Hangzhou has a wide variety of specialty delicacies, among which the most famous might be West Lake Vinegar Fish, renowned for its Zhejiang cuisine. In addition, there are West Lake Vinegar Fish, Zhejiang Longjing Shrimp, Zhejiang Snail Noodles, etc. If you want to try other Hangzhou delicacies, you can also tell me, and I can provide you with more information.",
"prompt": "The delicacies of Hangzhou include"
}
],
"stage": "pretrain",
"terminal_header": "### Pretrain Model\n\n\n$ python eval_model.py --load 0 --model 0\nMiniMind model parameter count: 104.03M(illion)\n[0] Auto test\n[1] Manual input\n0"
},
{
"arm": "original",
"pair_count": 8,
"pairs": [
{
"output": "As a chatbot, my existence is mainly to provide users with useful information, helping them solve problems and obtain information. I can answer a variety of common questions, provide various information, including history, culture, geography, science, technology, society, culture, etc. I can also learn and understand human language, language",
"prompt": "Please introduce yourself."
},
{
"output": "As an AI assistant, I don't have the ability to \"like\" like a human. My purpose is to answer questions raised by users and provide useful information and help. I am just a program designed to answer various questions and provide useful information. If you have any other questions, please let me know, and I will try my best to answer.\n\nLu Xun's \"Diary of a Madman\" reveals the hypocrisy and emptiness of feudal ethics. The novel, through the protagonist's inner monologue, reveals the hypocrisy and emptiness existing in society at that time. This contradiction and emptiness also led people to question and reflect on feudal ethics.\n\nAt the same time, the novel also reveals the influence and change of feudal ethics on society and individuals through the depiction of the characters' life experiences and inner worlds. This critical literary style and theme also made Lu Xun's literary works have a profound impact on society.\n\nIn summary, \"Diary of a Madman\" is a representative work of Lu Xun in literary creation. It criticizes the hypocrisy and emptiness of feudal ethics by depicting social problems at that time, while also reflecting various social issues and contradictions of that time.",
"prompt": "Which subject are you better at?"
},
{
"output": "First, you may have a cold, but this depends on your constitution, age, gender, condition, and medical history. If your constitution is weak, you may experience symptoms such as coughing and sore throat. In this case, it is recommended that you go to the hospital for an examination to determine the exact cause. The doctor may recommend that you use antibiotics or cold medicine to treat your symptoms. Based on the examination results, you can consider taking antibiotics, over-the-counter medications, or other drugs to relieve symptoms. In addition, you can also consider taking over-the-counter medications to relieve symptoms, such as antibiotics. However, it is best to consult a doctor to determine if it is suitable for your situation.",
"prompt": "I have been coughing for two weeks. Do I need to go to the hospital for a check-up?"
},
{
"output": "The speed of light refers to the speed at which light propagates in a vacuum, approximately 299,792,458 meters per second. This is a common unit of speed in physics. The speed of light is the most fundamental constant in nature, and its definition is that the speed of all matter in a vacuum exceeds one million kilometers per second. The speed of light is the propagation speed between matter in the universe, defined as the square of the speed of light in a vacuum divided by the square of the speed of light, that is, $c^2=m^2$. The absolute value of the speed of light is 2997*2458 meters per second, which is the speed of matter in a vacuum, approximately 4.08 meters per second. The speed of light has different meanings in different situations. For example, in the spectrum of a light source, the propagation speed of light is affected by the intensity and wavelength of light\n, while in the spectrum, the propagation speed of light is affected by information such as the intensity, frequency, brightness, and wavelength of light. The speed of light is constant in macroscopic objects, approximately 299,792,458 meters per second. In physics, the speed of light is constant, but for the propagation speed of light from a source, it is limited to a tiny range, even\nunable to transmit. The speed of light is the highest point of the source speed propagating in a vacuum, while the limitation of the source speed refers to the magnitude of the source speed. This limitation is caused by the relative positional relationship between the energy density of the source and the speed of light.",
"prompt": "Explain the physical concept of the speed of light in detail."
},
{
"output": "Hangzhou has many specialty delicacies, among which the more famous ones include:\n1. West Lake Vinegar Fish: Also known as \"Jiangsu fish,\" it is one of the traditional famous dishes of Hangzhou, with delicious soup and fresh carp.\n2. Crayfish: A traditional snack represented by crayfish, it is one of the unique snacks of Hangzhou.\n3. Hangzhou Xiaolongbao: Mainly made with tender meat filling, delicious soup, and fresh vegetables.\n4. Longjing Shrimp: Represented by fresh shrimp, it has a delicious seafood flavor and is also a very famous specialty snack of Hangzhou.\n5. Xitang: Famous for its ancient town, mainly made with fresh fish, paired with fresh peanuts and taro and other snacks, it is a very famous snack of Hangzhou.\nI hope the above recommendations are helpful to you.",
"prompt": "Recommend some specialty delicacies of Hangzhou."
},
{
"output": "A large language model refers to a neural network-based language model, mainly used for natural language processing tasks. It can perform tasks such as text summarization, machine translation, and dialogue systems. It can process large amounts of language data and automatically extract meaningful vocabulary, syntactic structures, and other information from it, providing support for fields such as text generation and speech recognition.",
"prompt": "Please explain the concept of \"large language model\" to me."
},
{
"output": "ChatGPT is a natural language processing tool developed by Google, designed to help users create meaningful conversations to understand natural language. It uses speech recognition technology, by analyzing the user's input language, it can understand the user's input, and use semantic analysis technology to help users answer various questions. It also supports text\ngeneration, can generate text, allowing users to create their own answers to understand the user's intent.",
"prompt": "How to understand ChatGPT?"
},
{
"output": "Introduce the history of the United States embers.",
"prompt": "Introduce the history of the United States, please."
}
],
"stage": "sft",
"terminal_header": "### SFT Model\n\n\n$ python eval_model.py --load 0 --model 1\nMiniMind model parameter count: 104.03M(illion)\n[0] Auto test\n[1] Manual input\n0"
},
{
"arm": "original",
"pair_count": 9,
"pairs": [
{
"output": "Hello! I am an AI language model capable of natural language conversation, answering questions, and generating text. I am a language model trained on large amounts of data that can understand and generate natural language.",
"prompt": "Please introduce yourself."
},
{
"output": "I am better at mathematics and logical reasoning. Although mathematics and logical reasoning are relatively abstract subjects, there is no necessary connection between them. Mathematics is a tool used to describe and solve problems, while logical reasoning is the process of reasoning and proving inferences. Mathematics is a foundational subject, and mathematics is the foundational subject of logic. Mathematics is a discipline that studies the concepts of quantity and space. Logical reasoning is a common branch of mathematics, mainly used for reasoning and proving the process of inference. Mathematicians use logical thinking to reason and prove the process of inference.",
"prompt": "Which subject are you better at?"
},
{
"output": "Lu Xun's \"Diary of a Madman\" has problems in criticizing feudal ethics, mainly because its language style and content are too exaggerated and hyperbolic, or do not conform to social reality, causing problems in society's evaluation of human nature and social relations.\n\nLu Xun's novels usually use a humorous, satirical, and ironic technique, revealing some problems of feudal society in a very humorous way, while also criticizing the constraints and rules of feudal ethics.\n\nIn criticizing feudal ethics, Lu Xun's novels usually use a very exaggerated and hyperbolic technique, revealing some problems in social reality in a very humorous way. This technique makes the work seem very absurd and hollow, while also criticizing the social order and moral concepts of the time.\n\nLu Xun's novels usually use a very humorous, satirical, and ironic technique, revealing some problems of feudal society in a very humorous way. This technique makes the work seem very absurd and hollow, while also criticizing social relations of the time, making the work seem very absurd and hollow.\n\nTherefore, in criticizing feudal ethics, Lu Xun's novels often use a humorous, satirical, and ironic technique, revealing social problems in a very humorous way. This technique makes the work seem very absurd and hollow, while also criticizing the social order and moral concepts of the time.",
"prompt": "How does Lu Xun's \"Diary of a Madman\" criticize feudal ethics?"
},
{
"output": "First, it is necessary to determine the specific situation of the cough. Coughing is a manifestation of physical discomfort, mainly due to physical discomfort, including lung infections, pneumonia, etc. If the cough has persisted for two weeks, it is recommended to go to the hospital for an examination. The doctor may examine the lungs to check for inflammation or infection. If not, the doctor may recommend examinations such as a chest X-ray or blood tests. If necessary, the doctor can also perform blood tests, throat radiation examinations, etc. During the examination, care should be taken to avoid excessive force, not to drink alcohol or eat irritating foods, and to avoid contact with others. If the cough persists for two weeks, it is recommended to go to the hospital promptly for diagnosis and treatment of the condition.",
"prompt": "I have had a cough for two weeks. Do I need to go to the hospital for a check-up?"
},
{
"output": "The speed of light refers to the propagation speed of light in a vacuum, and its unit of measurement is the speed of light (c). In a vacuum, the speed of light is 299,792,458 meters per second, approximately 393,458 meters per second.\n\nThe speed of light is one of the fundamental constants in physics, indicating that the speed of light is always equal to 299,792,458 meters per second at any time. In relativity, the speed of light is defined as 9.87924/966.71 seconds, while in other scientific fields, such as astronomy and aerospace, the speed of light is defined as 393,458 meters per second.\n\nIn summary, the speed of light is a very important concept because it is a fundamental concept in physics and is widely used in science and technology.",
"prompt": "Explain the physical concept of the speed of light in detail."
},
{
"output": "Hangzhou has many specialty foods. Here are a few recommendations for you: 1. West Lake Vinegar Fish: This is a delicate, sweet and sour specialty snack, a signature dish of Hangzhou. 2. Dongpo Pork: This is one of Hangzhou's traditional famous dishes, with tender meat and a fragrant, sticky texture. 3. Dongpo Steamed Bun: This is a traditional Chinese pastry, crispy in texture with rich fillings. 4. Handmade Dumplings: These are dumplings mainly filled with meat, with soft fillings and a refreshing taste. 5. Dongpo Soup: This is a traditional soup dish, rich in broth and delicious in flavor. 6. Xiaolongbao (Soup Dumplings): These are small steamed buns with rich, fragrant fillings and a chewy, springy texture. 7. Handmade Baozi (Steamed Buns): This is a traditional West Lake snack, with rich and delicious fillings. 8. Stone Pot Rice: This is a traditional snack mainly based on rice, with rich and delicious fillings. The above foods are all very distinctive. You can choose to try them according to your taste.",
"prompt": "Recommend some Hangzhou specialty foods."
},
{
"output": "A Large Language Model (Generative Area) is a neural network model trained on large-scale datasets that can handle various tasks of natural language, such as machine translation, text classification, sentiment analysis, speech recognition, and text generation. The principle of this model is through continuous iterative learning, constantly updating and optimizing the model, enabling the model to gradually understand the input natural language, thereby generating more accurate and natural text. Specifically, Generative adopts a model based on Recurrent Neural Networks (RNN). Through pre-training, it compares the input natural language sequence with the pre-trained sequence data, thereby obtaining more accurate and natural text output. Generative performs excellently in language generation, capable of generating more fluent and natural text. At the same time, Generative can continuously improve its generation effect through iterative learning and optimization, thereby better meeting user needs.",
"prompt": "Please explain the concept of \"Large Language Model\" to me."
},
{
"output": "ChatGPT is a model based on artificial intelligence technology. It can be used to complete natural language processing tasks, such as translation, question answering, generation, and classification. The accuracy of ChatGPT can reach over 90%, which means it can quickly understand the questions raised by users and give correct answers. It can also handle various complexities in natural language, such as grammar and semantics. ChatGPT can also handle text generation and classification tasks, such as text summarization, question answering, and classification. In addition, it can handle speech and image processing tasks for text, such as speech recognition and text-to-speech, as well as sentiment analysis tasks, such as comment and public opinion analysis. Overall, ChatGPT is a powerful natural language processing tool that can effectively process natural language and handle text and speech data.",
"prompt": "How to understand ChatGPT?"
},
{
"output": "Introduce the history of theUnitedStates, please.",
"prompt": "Introduce the history of the United States, please."
}
],
"stage": "dpo",
"terminal_header": "### DPO Model\n\n\n(base) root@C.26849627:~/minimind$ python eval_model.py --load 0 --model 2\nMiniMind model parameters: 104.03M (million)\n[0] Auto test\n[1] Manual input\n0"
},
{
"arm": "qk_norm_muon",
"pair_count": 7,
"pairs": [
{
"output": "It studies the influence of the bourgeoisie on social structure, mode of production, culture, morality, etc., from a metaphysical perspective. He believes that the bourgeoisie is the main driver of social structure, mode of production, culture, morality, etc., and its behavior and decisions directly affect the formation and change of social structure.\n\nMarxist theory still holds a very important position in the contemporary era. 1. What are the basic principles of Marxism?\n2. What is the object of study of Marxist theory?\n3. What does Marxist theory influence?\n4. Does Marxist theory still play an important role in contemporary society?",
"prompt": "Basic principles of Marxism"
},
{
"output": "What are they? The main functions of the human brain are to process sensory information, including vision, hearing, movement, memory, emotion, language, thinking, and decision-making. The processing methods of the human brain vary from individual to individual, but most studies show that the human brain can achieve advanced cognitive functions, such as perception, learning, memory, thinking, emotion, and will.",
"prompt": "Main functions of the human brain"
},
{
"output": "How is it calculated? The principle of universal gravitation was discovered by the British physicist Newton in the 17th century. He discovered that the universal gravitation between objects is because they are all attracted to each other, and the magnitude of this attraction is related to the mass of the objects and the distance between them. This law explains why the gravitational force between objects is proportional to their masses and inversely proportional to the square of the distance between them. Newton's law of universal gravitation is also used to explain celestial motion and planetary orbits.",
"prompt": "The principle of universal gravitation is"
},
{
"output": "Mount Everest, located in the Himalayas, with an altitude of 8848 meters. Mount Everest, located in the Himalayas, is the highest mountain in the world. Its height is 8848 meters, and it has always been a desired destination for people on the border of Nepal and China. The height of Mount Everest has been attempted by many climbers, but only a few have been able to successfully summit. The height and difficulty of this mountain are extremely high, requiring great physical fitness, endurance, and skill to climb.",
"prompt": "The highest mountain in the world is"
},
{
"output": "Will its concentration decrease as the temperature rises? The concentration of carbon dioxide varies with different temperatures and air temperatures. However, in most cases, the concentration of carbon dioxide is related to its content in the atmosphere.\n\nAt higher temperatures, the concentration of carbon dioxide may decrease because carbon dioxide is a greenhouse gas, and its concentration can be negligible. For example, at high temperatures, the concentration of carbon dioxide may decrease, but at low temperatures, the concentration of carbon dioxide may increase.\n\nFurthermore, under different temperatures and humidity levels, the concentration of carbon dioxide may change. For example, at higher temperatures, the concentration of carbon dioxide may increase because the rise in temperature leads to an increase in water vapor content, thereby causing the concentration of carbon dioxide in the air to decrease.\n\nIn summary, the concentration of carbon dioxide is affected by many factors, and being affected by temperature in the atmosphere will cause the concentration of carbon dioxide to decrease.",
"prompt": "Carbon dioxide in the air"
},
{
"output": "How many? The largest animal on Earth is the blue whale (Balaenoptera musculus). Blue whales can grow up to 30 meters long and weigh up to 173 tons. They are currently the largest known animals and the largest animals on Earth.\nBlue whales are distributed in all major oceans around the world, in the waters surrounding Antarctica. They are the largest species of whale, with a body weight that can reach 200 tons. Blue whales are carnivorous, feeding on small planktonic animals.\nBesides the blue whale, the largest animal on Earth is the African elephant (Loxodonta afasciola), which is the largest land animal on Earth. African elephants can grow up to 4 meters tall and weigh over 6 tons.\nIn summary, the blue whale is the largest animal in the world, and they live in the oceans around the globe.",
"prompt": "The largest animal on Earth has"
},
{
"output": "What dishes are worth trying?\nWhat is the answer to this question? My favorite is Longjing Shrimp, and I must eat it every time I go to Hangzhou!",
"prompt": "Hangzhou's cuisine has"
}
],
"stage": "pretrain",
"terminal_header": "### Pretrain Model\n\n\n$ CUDA_VISIBLE_DEVICES=1 python eval_model.py --load 0 --model 0\nMiniMind model parameters: 104.03M (million)\n[0] Auto test\n[1] Manual input\n0"
},
{
"arm": "qk_norm_muon",
"pair_count": 9,
"pairs": [
{
"output": "I am an AI language model capable of text generation and answering various questions. I am trained for natural language processing, language understanding, natural language generation, dialogue systems, etc., and can provide users with efficient, accurate, and convenient services. My goal is to provide users with accurate, comprehensive, and professional knowledge, helping them solve various problems and improve their own performance and abilities.",
"prompt": "Please introduce yourself."
},
{
"output": "I am better at English, mathematics, science, and history. In these subjects, I can use English and English grammar, learn English grammar and spelling rules, and be familiar with various historical events and cultures. I am also very good at English, mathematics, science, and history.",
"prompt": "Which subject are you better at?"
},
{
"output": "Lu Xun's \"Diary of a Madman\" is a literary work that opposes feudal ethics. In this work, the protagonist is a mental patient who is given a fate of self-destruction and begins to destroy himself. However, the novel also depicts the cruelty and depravity of feudal ethics. In the novel, Lu Xun uses his critique of feudal ethics to reflect on this system. He believes that feudal ethics is not only a critique of feudal ethics but also a rebellion against society. In the novel, Lu Xun's \"Diary of a Madman\" criticizes the oppression and enslavement of feudal ethics and raises questions about feudal ethics. He believes that feudal ethics is a corruption of human nature and an exploitation of feudal ethics. In addition, the novel also criticizes the critique of modern culture by feudal ethics. Lu Xun believes that feudal ethics is a kind of \"banned book,\" prohibiting the publication and dissemination of literary works and requiring public plagiarism. This behavior seriously violates the legitimacy of feudal ethics and severely damages the image of feudal ethics. In summary, Lu Xun's \"Diary of a Madman\" criticizes the oppression and enslavement of feudal ethics and raises questions about feudal ethics. This critique reflects the corruption and moral decay within Chinese feudal society, as well as the critique of modern culture by feudal ethics.",
"prompt": "How does Lu Xun's \"Diary of a Madman\" criticize feudal ethics?"
},
{
"output": "If the cough has persisted for two weeks, the doctor may recommend further examinations. These examinations include hearing tests, lung CT scans, and chest X-rays to assess the condition of the respiratory system. If your cough lasts more than two weeks, the doctor may suggest you go to the hospital for a visit.",
"prompt": "I have had a cough for two weeks. Do I need to go to the hospital for a check-up?"
},
{
"output": "The speed of light is one of the most fundamental physical constants in physics. It represents the speed at which light travels in a vacuum, approximately 299,792,458 meters per second. The speed of light is the speed at which light propagates in a vacuum, with a value of 299,792,458 meters per second.\nThe physical concept of the speed of light includes the following aspects:\n1. Relativity of the speed of light: The speed of light is a constant that is not affected by the observer's state of motion.\n2. Equivalence of the speed of light: The speed of light is not affected by the observer's state of motion and is a constant, meaning it is not affected by the observer's position.\n3. Conservation of light energy: The speed of light is invariant in any inertial reference frame, meaning it is not affected by the observer's state of motion.\n4. Propagation speed of light: The speed of light in a vacuum is 299,792,458 meters per second, approximately 299,792,458 meters per second.\n5. Invariance of the speed of light: The speed of light is not affected by the observer's state of motion and is a constant, meaning the speed of light in a vacuum is not affected by the observer's position.\nThe importance of the speed of light in physics cannot be ignored. It is not only a fundamental physical constant of the speed of light but also one of the most important fundamental constants in physics.",
"prompt": "Please provide a detailed introduction to the physical concept of the speed of light."
},
{
"output": "Hangzhou's food culture is very rich. Here are a few famous specialty dishes: 1. Longjing Shrimp: This is a traditional Hangzhou dish made with fresh shrimp as the main ingredient, processed through steaming, frying, boiling, and other steps, resulting in a delicious taste. 2. Dongpo Pork: This is a traditional dish made with pork as the main ingredient, slowly braised with seasonings such as scallions, ginger, garlic, star anise, and cinnamon. 3. West Lake Vinegar Fish: This is a sweet and sour dish made with fish as the main ingredient, seasoned with vinegar, sugar, salt, and other seasonings. 4. Beggar's Chicken: This is a traditional dish made with chicken as the main ingredient, slowly braised with seasonings such as Sichuan peppercorns, ginger, and scallions. 5. Dongpo Eggs: This is a traditional dish made with eggs as the main ingredient, slowly braised with seasonings such as scallions, ginger, and salt.",
"prompt": "Recommend some Hangzhou specialty foods."
},
{
"output": "A \"Large Language Model\" refers to a language model capable of understanding and generating natural language text. It is a natural language processing technology based on neural networks. Building on the previously mentioned language models, it can perform tasks such as text generation, machine translation, and question-answering systems. Training a large language model requires a large amount of data and computational resources, but the quality and reliability of its training results can directly affect its performance in natural language processing tasks.",
"prompt": "Please explain the concept of \"Large Language Model\" to me."
},
{
"output": "ChatGPT is an artificial intelligence language model that can answer questions based on an understanding of human language. It can understand and generate natural language, but it is not in the form of a \"chat.\" It is a computer program that can communicate using natural language like a human.\nTo understand ChatGPT, you need to use natural language processing (NLP) techniques and deep learning algorithms. These algorithms are trained to understand inputs and outputs and can generate human-like text. This may involve extracting features from text, such as grammar, syntax, and contextual information, and then using this information to generate new text.\nTo use ChatGPT, you need to provide it with a large amount of text data, which can be structured data (such as text documents or web pages) or unstructured data (such as images or audio). Then, you can use deep learning algorithms, such as Recurrent Neural Networks (RNNs) or Transformers, to train the model to generate human-like text.\nOverall, ChatGPT is a powerful tool that can help you process natural language, but understanding its complexity and abstraction remains a challenging task.",
"prompt": "How to understand ChatGPT?"
},
{
"output": "- Introduce the history of the United States.- Please.",
"prompt": "Introduce the history of the United States, please."
}
],
"stage": "sft",
"terminal_header": "### SFT Model\n\n\n$ python eval_model.py --load 0 --model 1\nMiniMind model parameters: 104.03M (million)\n[0] Auto test\n[1] Manual input\n0"
},
{
"arm": "qk_norm_muon",
"pair_count": 9,
"pairs": [
{
"output": "I am an AI assistant developed based on natural language processing technology. I can answer various questions and provide various services, including but not limited to providing information, providing entertainment, and helping to solve various problems. I am constantly learning and evolving to better serve users.",
"prompt": "Please introduce yourself."
},
{
"output": "I am better at computer science and mathematics. In these subjects, I am particularly good at programming, including Java and Python. I can use these programming languages to write code, develop software applications, and solve real-world problems.",
"prompt": "Which subject are you better at?"
},
{
"output": "Lu Xun's \"Diary of a Madman\" is an article that criticizes the corruption of human nature by feudal ethics. In the article, Lu Xun criticizes the rebellion and oppression of feudal ethics. He believes that the criticism of human nature by feudal ethics is wrong. First, feudal ethics is a feudal system that controls people's behavior and thoughts, limiting people's freedom and dignity. The criticism of human nature by feudal ethics is an oppression and exploitation of feudal ethics, which has had a negative impact on the progress of China's modernization process. Second, the criticism of human nature by feudal ethics is an oppression and exploitation of feudal ethics. The criticism of human nature by feudal ethics is an oppression and exploitation of feudal ethics. The oppression of feudal ethics by feudal ethics is an oppression and exploitation of feudal ethics. Finally, the criticism of human nature by feudal ethics expressed in Lu Xun's \"Diary of a Madman\" is an oppression and exploitation of feudal ethics. The criticism of human nature by feudal ethics is an oppression and exploitation of feudal ethics. In short, Lu Xun's \"Diary of a Madman\" criticizes feudal ethics, with the oppression and exploitation of feudal ethics as the main direction of criticism, and proposes rebellion and oppression against feudal ethics. At the same time, Lu Xun also reflects on the oppression and exploitation of feudal ethics, expressing reverence and awe for feudal ethics.",
"prompt": "How does Lu Xun's \"Diary of a Madman\" criticize feudal ethics?"
},
{
"output": "If you have been coughing for two weeks, it is recommended to go to the hospital for a check-up. Usually, it is best to have a check-up under the guidance of a doctor. The doctor will evaluate your symptoms and medical history and may perform some basic physical examinations, such as auscultation, facial examination, and blood tests. If your symptoms persist and do not show significant improvement, you may need further examination at the hospital. In some cases, the doctor may recommend blood tests, lung function tests, chest X-rays, electrocardiograms, etc. If your cough symptoms do not show significant improvement, you may need to go to the hospital for further examination. Regardless of the severity of your symptoms, the doctor will develop a treatment plan based on your condition. If your symptoms are severe or last for a long time, it is recommended to seek medical attention as soon as possible.",
"prompt": "I have had a cough for two weeks. Do I need to go to the hospital for a check-up?"
},
{
"output": "The speed of light is a commonly used concept in physics, referring to the speed at which light propagates in a vacuum. Its value is 299,792,458 meters per second, which is the fastest speed in nature. The physical concept of the speed of light includes the following points: 1. The speed of light is a natural constant, i.e., 299792.9835478 seconds. 2. The speed of light is a constant, i.e., the speed at which light propagates in a vacuum, which is the fastest speed in nature. 3. The speed of light is an important concept, significant for fields such as physics, astronomy, and cosmology. 4. The unit of the speed of light is meters per second, representing the speed at which light propagates in a vacuum, approximately 299,792,458 meters per second. 5. The speed of light is an important physical constant, closely related to the wave nature of light and electromagnetic radiation. 6. The speed of light plays an important role in relativity, being one of the foundations of general relativity. The speed of light is also one of the foundations of special relativity, an important concept for describing the relativity of object motion and light. In summary, the speed of light is a commonly used concept in physics, significant for fields such as physics, astronomy, and cosmology.",
"prompt": "Please provide a detailed introduction to the physical concept of the speed of light."
},
{
"output": "Hangzhou has many famous specialty foods, such as West Lake Vinegar Fish, Longjing Shrimp, Beggar's Chicken, Dongpo Pork, etc. In addition, Hangzhou's Xiaolongbao (soup dumplings), Glutinous Rice Chicken, Longjing Shrimp Sweet and Sour Fish, and other delicacies are also not to be missed.",
"prompt": "Recommend some Hangzhou specialty foods."
},
{
"output": "A large language model, also known as a large language model, is a language model based on a statistical model used to generate natural language text. It learns the patterns of language elements such as vocabulary, grammar, and context to build a model that can generate text conforming to language rules. Large language models have a wide range of applications, such as machine translation, speech recognition, and natural language generation.",
"prompt": "Please explain the concept of \"Large Language Model\" to me."
},
{
"output": "ChatGPT is an artificial intelligence language model that uses machine learning algorithms to generate human-like text. ChatGPT learns language patterns and grammatical rules by training on large amounts of text data, enabling it to generate human-like responses. It is an advanced technology widely used in various applications, such as chatbots, intelligent assistants, and language translation. To understand ChatGPT, it is necessary to delve into its internal working principles, such as how it generates responses based on previous input and how it uses probability-based statistical methods to learn language patterns and grammatical rules.",
"prompt": "How to understand ChatGPT?"
},
{
"output": "Introduce the history of the United States, please.",
"prompt": "Introduce the history of the United States, please."
}
],
"stage": "dpo",
"terminal_header": "### DPO Model\n\n\n$ python eval_model.py --load 0 --model 2\nMiniMind model parameter count: 104.03M(illion)\n[0] Automatic test\n[1] Manual input\n0"
}
],
"experiment": "8-3",
"output_count": 49,
"schema_version": "exp8-3-retained-outputs-v1",
"source_report": "chapter8/MiniMind-pretrain/README.md",
"source_report_sha256": "a8e1df1a9ee5cf013995e9ff3b963621485a838c97456b9f30e65ea9fdf55d50",
"stages": [
"pretrain",
"sft",
"dpo"
]
}
chapter8/MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/summary.json
+276
View File
@@ -0,0 +1,276 @@
{
"acceptance": {
"all_expected_outputs_retained": true,
"all_six_arm_stage_cells_retained": true,
"checkpoints_not_an_acceptance_artifact": true,
"eight_stage_balanced_blind_judgments": true,
"future_reproduction_commands_declared": true,
"historical_provenance_limitations_explicit": true,
"immutable_dataset_revision_lfs_hashes_and_sizes_frozen": true,
"immutable_source_revision_and_file_hashes_frozen": true,
"original_and_qk_norm_muon_compared": true,
"passed": true,
"pretrain_sft_and_dpo_compared": true,
"raw_historical_report_hashed": true,
"raw_judge_requests_responses_ids_usage_latency_retained": true,
"reported_loss_claims_qualified": true
},
"arm_averages": {
"original": {
"factuality": 1.375,
"instruction_following": 1.75,
"language_fluency": 3.0,
"overall": 2.0417
},
"qk_norm_muon": {
"factuality": 4.125,
"instruction_following": 3.0,
"language_fluency": 3.75,
"overall": 3.625
}
},
"experiment": "8-3",
"judge": {
"blind_seed": 730731,
"calls": 8,
"model": "doubao-seed-1-6-250615",
"provider": "ark",
"response_ids": [
"02178549583945856db6dee5d970b68ab3a378dc7e67e35390cf8",
"021785495839461d24b0b4d764165756d4018ab78dadfacc3782e",
"021785495839460982b29f8728970ed398f78ebc517c3ae19045a",
"021785495839459da6f4ee8e5443fabd9dcec9964b863512d1526",
"021785495872253a78e4bf9e905a6c19d35dac8ed03e6be35e22e",
"0217854958740424ebcb86fd234a9cf7780741b795c6db019ca22",
"0217854958861885bd2f0b9e6315ce45b86fb4e22ec749679fcc6",
"02178549588624031c26a7cf9ea611aa30972127acb8017b9f7cf"
],
"total_latency_ms": 285434.825,
"total_tokens": 15652
},
"limitations": [
"Historical checkpoints are intentionally not distributed and were not recreated in this audit.",
"The historical source revision, dataset byte identities, RNG state, and stepwise loss logs were not retained.",
"Frozen source/data revisions and the book lock define a future reproduction contract, not historical provenance.",
"The independent judge covers eight preregistered comparisons; all other retained outputs remain available for inspection.",
"The historical outputs are English translations in a bilingual report, so translation may affect the judge scores."
],
"per_case_arm_scores": {
"1": {
"original": {
"factual_errors": [
"In Nepali, its name means 'Goddess's Home' (incorrect; Nepali name Sagarmatha means 'Forehead of the Sky' or similar)"
],
"factuality": 3,
"instruction_following": 3,
"language_fluency": 3,
"rationale": "Correctly names Mount Everest, its location, and altitude, but contains a factual error about the Nepali name's meaning. Starts with an irrelevant 'which one?' (not following the prompt's statement structure) and has repetitive details (e.g., repeating altitude and location), leading to partial instruction following and flawed fluency."
},
"qk_norm_muon": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 5,
"language_fluency": 3,
"rationale": "Accurately identifies Mount Everest as the highest mountain, with correct location (Himalayas) and altitude (8848 meters). No material factual errors. Directly answers the prompt, though with repetitive phrasing (e.g., repeating 'Mount Everest, located in the Himalayas') and a fragmentary opening sentence, reducing fluency."
}
},
"2": {
"original": {
"factual_errors": [
"States CO2 concentration is 'about 20% of air' (actual ~0.04%)",
"Refers to CO2 as a 'very important element' (it is a compound)",
"Claims CO2 is 'main gas for respiration in humans' (humans exhale CO2, do not use it for respiration)"
],
"factuality": 0,
"instruction_following": 2,
"language_fluency": 3,
"rationale": "Contains severe factual errors (e.g., 20% concentration), misclassifies CO2 as an element, and misrepresents its role in respiration. Attempts to correct a false claim but introduces major inaccuracies. Language is coherent but flawed."
},
"qk_norm_muon": {
"factual_errors": [
"Claims CO2 concentration 'can be negligible' at higher temperatures due to being a greenhouse gas (false; greenhouse properties don't reduce concentration)",
"Contradicts temperature effect (higher temp 'may decrease' then 'may increase because... thereby causing decrease')",
"Incorrectly concludes temperature causes CO2 concentration to decrease (no evidence for direct relationship)"
],
"factuality": 0,
"instruction_following": 2,
"language_fluency": 3,
"rationale": "Contains multiple contradictory and false claims about CO2 concentration and temperature; attempts to address the topic but with major errors. Language is understandable but has logical defects."
}
},
"3": {
"original": {
"factual_errors": [
"Claims matter exceeds speed of light",
"Incorrect formula $c^2=m^2$",
"Miscalculated speed (2997*2458 and 4.08 m/s)",
"States speed depends on light intensity",
"Falsely claims light propagation is 'limited to a tiny range'"
],
"factuality": 0,
"instruction_following": 1,
"language_fluency": 2,
"rationale": "Contains numerous material falsehoods about speed of light values, matter speed, and propagation; largely incoherent and fails to explain the concept."
},
"qk_norm_muon": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 4,
"language_fluency": 4,
"rationale": "Accurately states the speed of light in a vacuum and key concepts like invariance; minimal redundancy but no material factual errors."
}
},
"4": {
"original": {
"factual_errors": [
"Claims ChatGPT is 'developed by Google' (it is developed by OpenAI)",
"States it 'uses speech recognition technology' (ChatGPT is primarily text-based, not focused on speech recognition)"
],
"factuality": 0,
"instruction_following": 3,
"language_fluency": 3,
"rationale": "Contains major factual errors about developer and core technology. Attempts to explain functionality but is incomplete (e.g., mid-sentence line break) and has redundant phrasing, making it partially understandable but flawed."
},
"qk_norm_muon": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 5,
"language_fluency": 5,
"rationale": "Accurately identifies ChatGPT as an AI language model using NLP and deep learning (e.g., Transformers), explains training on text data. No material factual errors. Directly answers the task by explaining what it is and how it works. Fluent and coherent throughout."
}
},
"5": {
"original": {
"factual_errors": [
"mentions 'embers' which is irrelevant to and not part of US history"
],
"factuality": 0,
"instruction_following": 0,
"language_fluency": 3,
"rationale": "Output repeats the prompt with the irrelevant term 'embers', containing a material factual error, failing to introduce US history, and is understandable but nonsensical due to 'embers'."
},
"qk_norm_muon": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 0,
"language_fluency": 3,
"rationale": "Output repeats the prompt without adding content, containing no material factual errors, failing to introduce US history, and is understandable but has awkward punctuation."
}
},
"6": {
"original": {
"factual_errors": [
"Incorrect approximate speed (393,458 m/s instead of ~299,792,458 m/s)",
"Nonsensical definition in relativity: '9.87924/966.71 seconds' (speed cannot be defined in seconds)",
"Incorrect unit statement: 'unit of measurement is the speed of light (c)' (unit should be meters per second)"
],
"factuality": 1,
"instruction_following": 2,
"language_fluency": 3,
"rationale": "Contains multiple severe factual errors: incorrect speed values, nonsensical definitions in relativity, and wrong unit description. Partially addresses the concept but is undermined by critical inaccuracies."
},
"qk_norm_muon": {
"factual_errors": [
"Incorrect unit in point 1: '299792.9835478 seconds' (seconds is a unit of time, not speed)"
],
"factuality": 3,
"instruction_following": 3,
"language_fluency": 2,
"rationale": "Has a notable unit error but correctly states the speed value. Repeats points excessively but provides more relevant details (relativity, scientific fields) than A."
}
},
"7": {
"original": {
"factual_errors": [
"Claims 'accuracy can reach over 90%' (no general 'accuracy' metric exists for ChatGPT; performance varies by task and lacks substantiation); incorrectly states it 'handles speech and image processing tasks' (ChatGPT is primarily text-based, with no native image processing capabilities)"
],
"factuality": 2,
"instruction_following": 3,
"language_fluency": 4,
"rationale": "Contains material factual errors (unsubstantiated accuracy claim, image processing misstatement); explains basic function/uses but not 'how it works'; text is coherent with minor repetition."
},
"qk_norm_muon": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 5,
"language_fluency": 5,
"rationale": "No material factual errors; accurately explains ChatGPT as an AI language model using ML algorithms, training on text data to learn patterns, and working principles (response generation via input and probability-based methods); fully addresses 'what it is' and 'how it works' with coherent, natural language."
}
},
"8": {
"original": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 0,
"language_fluency": 3,
"rationale": "Output repeats the prompt without introducing U.S. history (non-answer, instruction following 0). No factual content (no material errors, factuality 5). Language has a typo ('theUnitedStates' missing space), understandable with defects (fluency 3)."
},
"qk_norm_muon": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 0,
"language_fluency": 5,
"rationale": "Output repeats the prompt without introducing U.S. history (non-answer, instruction following 0). No factual content (no material errors, factuality 5). Language is coherent and natural (fluency 5)."
}
}
},
"retained": {
"cells": 6,
"outputs": 49,
"selected_comparisons": 8
},
"schema_version": "exp8-3-summary-v1",
"scientific_findings": {
"blind_judge_overall_delta_qk_norm_muon_minus_original": 1.5833,
"blind_judge_prefers_qk_norm_muon_overall": true,
"reported_loss_comparison_retained_but_not_independently_recomputed": true,
"wins": {
"original": 0,
"qk_norm_muon": 7,
"tie": 1
}
},
"stage_averages": {
"original": {
"dpo": {
"factuality": 2.6667,
"instruction_following": 1.6667,
"language_fluency": 3.3333,
"overall": 2.5556
},
"pretrain": {
"factuality": 1.5,
"instruction_following": 2.5,
"language_fluency": 3.0,
"overall": 2.3333
},
"sft": {
"factuality": 0.0,
"instruction_following": 1.3333,
"language_fluency": 2.6667,
"overall": 1.3333
}
},
"qk_norm_muon": {
"dpo": {
"factuality": 4.3333,
"instruction_following": 2.6667,
"language_fluency": 4.0,
"overall": 3.6667
},
"pretrain": {
"factuality": 2.5,
"instruction_following": 3.5,
"language_fluency": 3.0,
"overall": 3.0
},
"sft": {
"factuality": 5.0,
"instruction_following": 3.0,
"language_fluency": 4.0,
"overall": 4.0
}
}
},
"status": "passed"
}
chapter8/MiniMind-pretrain/validation/runs/exp8-4-training-report-20260731-v1/manifest.json
+83
View File
@@ -0,0 +1,83 @@
{
"acceptance": {
"all_64_historical_outputs_retained": true,
"all_eight_configuration_cells_retained": true,
"checkpoints_not_an_acceptance_artifact": true,
"eight_image_aware_arm_blind_judgments": true,
"future_reproduction_commands_declared": true,
"historical_provenance_limitations_explicit": true,
"historical_report_content_hashed": true,
"immutable_dataset_clip_and_eval_image_inputs_frozen": true,
"immutable_original_and_improved_source_revisions_frozen": true,
"passed": true,
"raw_judge_requests_responses_ids_usage_latency_retained": true,
"request_images_match_pinned_sha256": true,
"same_eight_images_present_in_every_cell": true
},
"artifacts": [
{
"bytes": 22282,
"path": "retained_outputs.json",
"sha256": "6333571fcdb95a84bece9fdf65456f328e7e048342ede37f71e3e03c68c38df6"
},
{
"bytes": 10758,
"path": "reproduction_contract.json",
"sha256": "e530aa559f940a2034e79315432789bf017abf0fc247adba3fb5e802893ea1d7"
},
{
"bytes": 2596470,
"path": "judge_receipts.json",
"sha256": "c777a01a7a484d7a6184575be101c167c0007442144e2bcb1f7047cee0dbc5f0"
},
{
"bytes": 33310,
"path": "summary.json",
"sha256": "fa904dfee4e8dbdf0c9986db79b3eba18adc00b3bfd823e9a99fca10e69a3f6d"
},
{
"bytes": 2116,
"path": "report.md",
"sha256": "0906af01fcf1f6c11528215fe8767c6fd5cad0743286cb0cf94af0df300f3df6"
}
],
"checkpoint_policy": "not distributed; not an acceptance artifact",
"created_at": "2026-08-17T05:34:53.397311+00:00",
"experiment": "8-4",
"inputs": [
{
"bytes": 155255,
"path": "chapter8/MiniMind-pretrain/README.md",
"sha256": "a8e1df1a9ee5cf013995e9ff3b963621485a838c97456b9f30e65ea9fdf55d50"
},
{
"bytes": 14150,
"path": "pyproject.toml",
"sha256": "33cc27c2759f353663d6907f1a918a41a49fb8d3e4e28370d508136449c3b156"
},
{
"bytes": 1311181,
"path": "uv.lock",
"sha256": "347fc87f40526372c284c61e5374536ce8d2071936cef8844d350470ecdf0d0b"
},
{
"bytes": 42148,
"path": "chapter8/MiniMind-pretrain/validation/run_vlm_training_report_audit.py",
"sha256": "d59e8175ce19a1973005263f6f7e4642e1ee11402860b4fadf6a1fd192d4cc7d"
},
{
"bytes": 14797,
"path": "chapter8/MiniMind-pretrain/validation/validate_vlm_evidence.py",
"sha256": "4ce53d7000d7f45a8e8012db1989374fcf42f5f832fbe134aca098e826b95db5"
},
{
"bytes": 6800,
"path": "chapter8/MiniMind-pretrain/validation/test_vlm_training_report_audit.py",
"sha256": "6259638aae28d1035f9d597f8f5ea17c7a66bd6ba8e3759ed1c842c94c19d1e1"
}
],
"run_dir": "validation/runs/exp8-4-training-report-20260731-v1",
"run_id": "exp8-4-training-report-20260731-v1",
"schema_version": "exp8-4-manifest-v1",
"status": "passed"
}
chapter8/MiniMind-pretrain/validation/runs/exp8-4-training-report-20260731-v1/report.md
+26
View File
@@ -0,0 +1,26 @@
# Experiment 8-4 retained-training-report audit
## Result
Status: **passed**. The historical report retains 64 image descriptions across 8 configurations and the same 8 images. Each image was inspected by a real image-capable ARK judge together with all eight arm-blind captions.
| Configuration | Grounding | Hallucination control | Coverage | Specificity | Overall | Best count |
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
| without_muon_sft | 2.1250 | 1.3750 | 2.3750 | 1.7500 | 1.9062 | 1 |
| without_muon_pretrained | 1.8750 | 2.6250 | 1.5000 | 0.8750 | 1.7188 | 1 |
| muon_from_dpo_sft | 1.3750 | 1.1250 | 1.8750 | 1.7500 | 1.5312 | 2 |
| muon_from_sft_pretrained | 1.8750 | 2.0000 | 1.3750 | 0.8750 | 1.5312 | 2 |
| muon_from_sft_sft | 1.2500 | 1.1250 | 1.5000 | 1.2500 | 1.2812 | 1 |
| muon_from_pretrain_sft | 1.1250 | 0.5000 | 1.3750 | 1.1250 | 1.0312 | 0 |
| muon_from_dpo_pretrained | 1.0000 | 1.5000 | 0.8750 | 0.3750 | 0.9375 | 0 |
| muon_from_pretrain_pretrained | 0.7500 | 1.3750 | 0.7500 | 0.6250 | 0.8750 | 1 |
The highest descriptive judge mean was **without_muon_sft** at **1.9062**. Averaged across all four base configurations, full VLM SFT changed the score by **+0.1718** versus projection-only pretraining.
The isolated report comparison pairs original/SFT-base against QK-Norm+Muon/SFT-base at each VLM stage. QK-Norm and Muon still change together, so no Muon-only causal claim is made. All author-written qualitative claims remain historical observations rather than pass/fail gates.
## Provenance and reproduction boundary
`reproduction_contract.json` freezes separate pre-QK-Norm and QK-Norm+Muon MiniMind-V revisions, the corresponding base-LLM revisions, script-compatible VLM dataset Git-LFS objects, the CLIP weight object, all eight evaluation-image hashes, and future commands. These pins are not misrepresented as the historical checkout.
Training checkpoints remain local by book policy and are not acceptance artifacts. The accepted artifact is this content-hashed report, all 64 retained outputs, eight raw image-aware judge receipts, and explicit limitations.
chapter8/MiniMind-pretrain/validation/runs/exp8-4-training-report-20260731-v1/reproduction_contract.json
+178
View File
@@ -0,0 +1,178 @@
{
"checkpoint_policy": {
"acceptance_artifact": false,
"distributed_with_book": false,
"reason": "Training checkpoints are intentionally not distributed to readers.",
"required_artifact": "reproducible evidence-backed training report"
},
"experiment": "8-4",
"future_reproduction": {
"base_llm_source": {
"dependency": "Use the Experiment 8-3 data/commands to produce original-SFT and improved pretrain/SFT/DPO 768-dimension base checkpoints.",
"original_files_sha256": {
"eval_model.py": "b9f7ea9d7f517551362bbf2da8f1de006b8c734bcba774b2be752bc63cc4349d",
"model/model_minimind.py": "7cb069cb0cb0dfa123cf11ea394d0001270bc683c0a2dfe4120fc3b861ffc0a4",
"trainer/train_dpo.py": "5e556a3089e43681638cdbf5adafb9d085bb1de5e4ea8da3ee522dfae02e3599",
"trainer/train_full_sft.py": "a57422f1df80bf2867f31f3b4a646a92ac7f66729e98a3b32cd1ec4d6780cb8b",
"trainer/train_pretrain.py": "ddd122645a9f1043bc8dac69a81ac51d2df95df8745d25faed7963d38fedc328"
},
"original_revision": "6d160ea20b98324632c4447ee63ec7cfa9becd20",
"qk_norm_muon_files_sha256": {
"eval_model.py": "43930a4b55048a4a3ffa17eb78ae67d59582d639aa9365f0bbf41ba149128af8",
"model/model_minimind.py": "2d33988711c704be6a22c4c61489b23106a2340a7cb8b97ebe3e40f30819cbb0",
"trainer/train_dpo.py": "97f2c31cc8bc21a777e2efcb5e2fa35a49e4e9e3698db120148f8a0b2f678449",
"trainer/train_full_sft_muon.py": "acd0b7db5b1d8b25d3c3103f92d68a7d381f9322a1b33bbec34be3d005930bad",
"trainer/train_pretrain_muon.py": "fc83d07754ec3a8c156b6b8bfc0fd4326edecb72efabc5e08ae4ff5e3a7029bc"
},
"qk_norm_muon_revision": "8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795",
"repository": "bojieli/minimind"
},
"commands": {
"dataset": "git clone https://huggingface.co/datasets/jingyaogong/minimind-v_dataset dataset-source && git -C dataset-source checkout --detach ac9d03a3fd26a2d8e74bda374d9a2ddba49e4c1b && cp dataset-source/{pretrain_data.jsonl,sft_data.jsonl} dataset/ && unzip dataset-source/pretrain_images.zip -d dataset && unzip dataset-source/sft_images.zip -d dataset",
"evaluation": "For every isolated out_dir, preserve both checkpoints, copy the selected *_muon_768.pth name to eval_vlm.py's pretrain_vlm_768.pth or sft_vlm_768.pth compatibility name when needed, then run python eval_vlm.py --load 0 --model_mode 0 and --model_mode 1 on the eight hash-pinned images with seed 1337.",
"improved_matrix": "For each BASE in pretrain,sft,dpo, install the corresponding Experiment-8-3 QK-Norm+Muon 768-dimension checkpoint as runs/muon-from-$BASE/out/llm_768.pth, then run train_pretrain_vlm_muon.py and train_sft_vlm_muon.py with the same four-epoch data arguments in that isolated out_dir.",
"improved_source": "git clone https://github.com/bojieli/minimind-v.git sources/qk-norm-muon-minimind-v && git -C sources/qk-norm-muon-minimind-v checkout --detach ead791c530fa5f9a3549dbfe9e11ec732d18d2e5",
"original_pretrain_vlm": "install -m 0644 <exp8-3-original-sft-768.pth> runs/original/out/llm_768.pth && cd trainer && torchrun --nproc_per_node=8 train_pretrain_vlm.py --out_dir ../runs/original/out --epochs 4 --hidden_size 768 --num_hidden_layers 16 --data_path ../dataset/pretrain_data.jsonl --images_path ../dataset/pretrain_images --use_wandb",
"original_sft_vlm": "cd trainer && torchrun --nproc_per_node=8 train_sft_vlm.py --out_dir ../runs/original/out --epochs 4 --hidden_size 768 --num_hidden_layers 16 --data_path ../dataset/sft_data.jsonl --images_path ../dataset/sft_images --use_wandb",
"original_source": "git clone https://github.com/bojieli/minimind-v.git sources/original-minimind-v && git -C sources/original-minimind-v checkout --detach 765908051d0837d60cecfb93f8390334e2e55f1e",
"vision_encoder": "git clone https://huggingface.co/openai/clip-vit-base-patch16 model/vision_model/clip-vit-base-patch16 && git -C model/vision_model/clip-vit-base-patch16 checkout --detach 57c216476eefef5ab752ec549e440a49ae4ae5f3"
},
"environment": {
"book_lock": "uv.lock",
"book_pyproject": "pyproject.toml",
"boundary": "The book lock freezes a future Python environment; CUDA, drivers, and the historical GPU image were not retained.",
"install": "uv sync --locked --python 3.12 --extra ch7 --extra dev"
},
"evaluation_images": {
"Astronaut-Space.jpg": {
"sha256": "f466cdafecbdb85d2bad586896db5db3313afe18f9b3505667756cd25b747747",
"source_filename": "太空宇航员-Astronaut-Space.jpg"
},
"Bicycle-Flowers.jpg": {
"sha256": "44fae0fafcd52c20b9bcaded897facbff00f61019cdd0aea543addf8499ad899",
"source_filename": "自行车鲜花-Bicycle-Flowers.jpg"
},
"Chair-Elderly-Reading.jpg": {
"sha256": "8fe91a90e837c33230d21cfe7ba5020e71b3ae99ac4c3fbd6d32cb54f51def53",
"source_filename": "椅子老人看书-Chair-Elderly-Reading.jpg"
},
"Dog-Woman-Sea.jpg": {
"sha256": "ba90d8b8738a44eac70811be5c89f767492b167ad4f6f6c31aa4591837d7e3dc",
"source_filename": "小狗美女海边-Dog-Woman-Sea.jpg"
},
"Panda-Grassland.jpg": {
"sha256": "0b7610a881039f0effdbfa46e9bb189132443d3ce2956856e8adf66d1ca22f8c",
"source_filename": "熊猫草地-Panda-Grassland.jpg"
},
"Rainbow-Falls.jpg": {
"sha256": "1c8b74debaceb2e0bb6171b182084afe49288a0cc8089eb91eac69d067c27b10",
"source_filename": "彩虹瀑布-Rainbow-Falls.jpg"
},
"city-traffic.jpg": {
"sha256": "73e90d82fbc5b1cf43b40de782b443f93f43a34e66b8ddebf3146d5dc1f83e00",
"source_filename": "城市车水马龙-city-traffic.jpg"
},
"dance.jpg": {
"sha256": "939e3132c8d3aec81f66f8aa928b476aaa25e00d94f1097f4974e73c913d5d8c",
"source_filename": "舞蹈-dance.jpg"
}
},
"vision_encoder": {
"file": {
"bytes": 598641023,
"lfs_sha256": "ec89c7b09c749a60aae3c9cd910516f24b58214a7df060b48962d14c469cfbf0",
"path": "pytorch_model.bin"
},
"repository": "openai/clip-vit-base-patch16",
"revision": "57c216476eefef5ab752ec549e440a49ae4ae5f3"
},
"vlm_dataset": {
"files": {
"pretrain_data.jsonl": {
"bytes": 134315765,
"lfs_sha256": "abc9f2ba44190646692fbe7e2b49c366c5045490989fb32d2c5e960dd0ee10e4"
},
"pretrain_images.zip": {
"bytes": 2614907051,
"lfs_sha256": "64d56cee145bed75bc7f94c9cbf58882c41c4a0fea993014e27de7490b49e8b7"
},
"sft_data.jsonl": {
"bytes": 173137988,
"lfs_sha256": "c1993d38c3a22a8bdfee65affc82d6559e5bb62e785b0f21c9151c75116151fc"
},
"sft_images.zip": {
"bytes": 1026332147,
"lfs_sha256": "89ee34facc6793c51613613e0b10cac078942282f5fdec48d85751c6224bc3c2"
}
},
"repository": "jingyaogong/minimind-v_dataset",
"revision": "ac9d03a3fd26a2d8e74bda374d9a2ddba49e4c1b",
"selected_for_jsonl_script_compatibility": true
},
"vlm_source": {
"not_claimed_as_historical_revisions": true,
"original_files_sha256": {
"dataset/lm_dataset.py": "df20d57460d2845841ddf2e0faced1af1f7ec169e7fda3cd50fe3b2854288a92",
"eval_vlm.py": "9d883e4adbab0a7b88fd0cb9034132559a365387ec273ac4811cdd5ad28d5cda",
"model/model_minimind.py": "105429e93dcbe87145264d72d46a6add7639666036e999628c76ae50582507dc",
"model/model_vlm.py": "4ee42b298db68f30fbfa06d7686aa375d41a697628c770d0a134bca40ca9ea80",
"model/tokenizer.json": "d98595c6aef70d95f72748582fb9b4f53d76dd58c1ae1dd702ad7c84e1caf5e4",
"model/tokenizer_config.json": "dbbdb7eea33aba5c2608471494c93f650a2cf46fbe4a7489e531537ddadee746",
"requirements.txt": "a9bddf49d3ccbc9f8a2508ea039aebc0b996dccb0d3618d1b119af53a5d49869",
"trainer/train_pretrain_vlm.py": "4d30d54a940ae2eced204971cc03aafb3eb41a5c84c9f033cfdf162e63924a4d",
"trainer/train_sft_vlm.py": "8e3b920a6a135eb126bbeea07e2db748729cdd86050925b282a80537bc324e5f"
},
"original_revision": "765908051d0837d60cecfb93f8390334e2e55f1e",
"qk_norm_muon_files_sha256": {
"dataset/lm_dataset.py": "df20d57460d2845841ddf2e0faced1af1f7ec169e7fda3cd50fe3b2854288a92",
"eval_vlm.py": "9d883e4adbab0a7b88fd0cb9034132559a365387ec273ac4811cdd5ad28d5cda",
"model/model_minimind.py": "4771bc4b2ac367a6e6415c42c30bcdb54bec0397708f87de3c390042680b1e9e",
"model/model_vlm.py": "4ee42b298db68f30fbfa06d7686aa375d41a697628c770d0a134bca40ca9ea80",
"model/tokenizer.json": "e489029175fb3f94b8211a120a72a2ee41a664db65b828d077c7bde989c845a9",
"model/tokenizer_config.json": "190cc4738bac3b6f6b563376019c581b320fdb0260a03b9d5ab806296c8c6bb8",
"requirements.txt": "a9bddf49d3ccbc9f8a2508ea039aebc0b996dccb0d3618d1b119af53a5d49869",
"trainer/muon.py": "00c2c6a225edeb55433df0724c3c74f6ff98ac4b2cc73c4aafcff686824f6267",
"trainer/train_pretrain_vlm_muon.py": "f39af354c588747d9d5e522c9374a7f59a35d57aa649da74957da67d95d25bc6",
"trainer/train_sft_vlm_muon.py": "1fd56b3e8bed2714b4d10ceba5d57ada0b95d00dfbfb514481498fef0c0dd03d"
},
"qk_norm_muon_revision": "ead791c530fa5f9a3549dbfe9e11ec732d18d2e5",
"repository": "bojieli/minimind-v"
}
},
"historical_evidence_boundary": {
"claim": "The author's report establishes that eight VLM configurations were evaluated on eight images. It does not establish byte identity of the historical code, datasets, base/VLM checkpoints, or RNG state.",
"eight_historical_cells_and_64_outputs_retained": true,
"historical_base_checkpoint_hashes_retained": false,
"historical_dataset_hashes_retained": false,
"historical_rng_and_stepwise_logs_retained": false,
"historical_source_revisions_retained": false,
"historical_training_executed": true,
"historical_vlm_checkpoint_hashes_retained": false
},
"reported_training_design": {
"base_llm_stages": [
"pretrain",
"sft",
"dpo"
],
"parameter_count_millions": {
"original": 104.622,
"qk_norm_muon": 104.625
},
"projection_pretraining_freezes_llm": true,
"reported_epochs": 4,
"seed_in_current_source": 1337,
"sft_unfreezes_full_model": true,
"source_verified_mechanisms": {
"improved_revision_has_qk_norm_before_rope": true,
"improved_revision_uses_muon_for_selected_2d_weights": true,
"original_revision_precedes_qk_norm_commit": true,
"vision_encoder_is_frozen_clip": true
},
"vlm_stages": [
"pretrained",
"sft"
]
},
"schema_version": "exp8-4-reproduction-contract-v1"
}
chapter8/MiniMind-pretrain/validation/runs/exp8-4-training-report-20260731-v1/retained_outputs.json
+366
View File
@@ -0,0 +1,366 @@
{
"cell_count": 8,
"cells": [
{
"architecture": "original",
"base_llm_stage": "sft",
"config": "without_muon_pretrained",
"historical_command": "$ python eval_vlm.py --load 0 --model 0",
"output_count": 8,
"outputs": [
{
"image": "Rainbow-Falls.jpg",
"output": "Water droplets on the rainbow bridge"
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "A young person decorating the beach with a dog's head."
},
{
"image": "dance.jpg",
"output": "This dancer is performing on stage."
},
{
"image": "Astronaut-Space.jpg",
"output": "An astronaut is performing a spacewalk in space."
},
{
"image": "city-traffic.jpg",
"output": "On the city streets at nightfall."
},
{
"image": "Panda-Grassland.jpg",
"output": "In the zoo, a panda is eating bamboo."
},
{
"image": "Bicycle-Flowers.jpg",
"output": "Using old bicycle tires and wheels on an old bicycle, adding flowers and decorations to the bicycle."
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "An elderly woman reading a book in the park."
}
],
"vlm_stage": "pretrained"
},
{
"architecture": "original",
"base_llm_stage": "sft",
"config": "without_muon_sft",
"historical_command": "$ python eval_vlm.py --load 0 --model 1",
"output_count": 8,
"outputs": [
{
"image": "Rainbow-Falls.jpg",
"output": "This picture depicts a huge, colorful giant waterfall located next to a highway. From the image, you can see a massive and majestic giant waterfall. A giant rainbow flag hangs above the waterfall, adding visual appeal. The waterfall is surrounded by beautiful scenery, adding beauty to this picturesque landscape."
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "This image depicts a woman sitting on a beach with her dog beside her, while her dog sits on a blue and white checkered blanket."
},
{
"image": "dance.jpg",
"output": "The picture depicts a woman performing a dance on stage, her dance steps soaring high above the stage. Several chairs are placed on the stage, one on the left side of the frame, another on the right, and one more on the right side.\nMultiple chairs are placed on the stage, one near the left, another on the right. There is also a clock on the stage, possibly to help the performer keep time."
},
{
"image": "Astronaut-Space.jpg",
"output": "The image shows a huge spaceship that looks like a giant commercial airplane. This airplane is in the center of the frame, flying over a small hill and a barn. The airplane looks like a hangar, filled with all sorts of things.\nAround the airplane, some people might be observing this large commercial aircraft. Some of them seem to be watching the pilot's actions, while others might be admiring the details of the aircraft or enjoying the scene."
},
{
"image": "city-traffic.jpg",
"output": "This picture depicts a busy city street with heavy traffic, several cars and a truck moving. Tall buildings and streetlights illuminate the area, creating a vibrant and lively atmosphere.\nOn the street, various vehicles can be seen, including several cars and a truck. Some pedestrians also appear, possibly going about their daily activities or navigating the urban environment.\nOverall, this image captures a vibrant urban environment where vehicles, pedestrians, and traffic lights together create a lively atmosphere."
},
{
"image": "Panda-Grassland.jpg",
"output": "The image depicts a black and white panda wearing glasses, a long-haired black and white panda sitting on the grass. This panda seems to be staring at the camera, it looks like it is looking at the camera's image. This panda seems to be lying on the grass resting, possibly to avoid the sun or rain."
},
{
"image": "Bicycle-Flowers.jpg",
"output": "The image shows a bicycle parked outside a building, with a row of vases and bouquets visible in the background. The bicycle occupies a considerable portion of the frame, extending from left to right. Throughout the scene, there are three potted plants, one near the left, another on the right, and a third on the left.\nBesides the bicycle, there are two people in the scene, one standing on the right and the other near the left side of the image."
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "The image depicts an elderly man wearing glasses sitting on a park bench. He seems to be reading a book, possibly enjoying the outdoor environment. Several cars are parked nearby, one in the center of the image and another on the right. There is also a bench in the background, right next to the person sitting on the bench."
}
],
"vlm_stage": "sft"
},
{
"architecture": "qk_norm_muon",
"base_llm_stage": "dpo",
"config": "muon_from_dpo_pretrained",
"historical_command": "$ python eval_vlm.py --load 0 --model 0",
"output_count": 8,
"outputs": [
{
"image": "city-traffic.jpg",
"output": "Night streets with elevators and sidewalks."
},
{
"image": "Astronaut-Space.jpg",
"output": "Spaceman sent to a new Earth"
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "You have a small ball in your palm, you throw it into the water."
},
{
"image": "Rainbow-Falls.jpg",
"output": "Viewed on the water, water landscape"
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "An elderly person sitting on a park bench."
},
{
"image": "Panda-Grassland.jpg",
"output": "This photo captures the cuteness and loveliness of this little white bear."
},
{
"image": "Bicycle-Flowers.jpg",
"output": "Riding a bicycle on the street, bicycles everywhere, I saw many cars, this is my bicycle"
},
{
"image": "dance.jpg",
"output": "Performed by actors performing on stage"
}
],
"vlm_stage": "pretrained"
},
{
"architecture": "qk_norm_muon",
"base_llm_stage": "dpo",
"config": "muon_from_dpo_sft",
"historical_command": "$ python eval_vlm.py --load 0 --model 1",
"output_count": 8,
"outputs": [
{
"image": "city-traffic.jpg",
"output": "This picture depicts a busy city street filled with skyscrapers and vehicles. At least twelve cars are driving on the street, some parked on the side, while others are moving.\nSeveral traffic lights can be seen, one on the left side of the street, another in the middle, and one on the right. Additionally, a bus is heading towards the center of the frame, adding to the lively atmosphere of this bustling city."
},
{
"image": "Astronaut-Space.jpg",
"output": "The person in the picture is wearing a black spacesuit and glasses, sitting on what looks like a space shuttle. They seem to be looking at a space shuttle flying in front of them, which is located on the right side of the scene.\nBesides the space shuttle, there are two other small boats in the picture, one on the left and another on the right. A smaller plane can also be seen in the background, and the person is located on the far right."
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "The image depicts a woman sitting on a bench by the beach, holding a small dog outside to sunbathe. She is wearing a dress and seems to be enjoying the outdoors.\nIn the background, several other people can be seen scattered on the beach, some closer to the water and others farther away. They all look happy, possibly spending a good time with this woman."
},
{
"image": "Rainbow-Falls.jpg",
"output": "The image shows a beautiful fountain surrounded by mist above it. The fountain is located above a high mountain, adding a magnificent background to the scene. An umbrella can be seen on the water, providing shade for visitors.\nThe terrain around the fountain consists of hills and mountains, offering a picturesque landscape for visitors. The presence of the fountain, water features, and surrounding mountains creates a serene and awe-inspiring atmosphere, making the area an ideal resort destination for photography enthusiasts and outdoor lovers."
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "The image shows an elderly man sitting on a park bench, sitting in front of a book, holding it in his hands. The person looks relaxed and casual, possibly enjoying the outdoors.\nThere are several trees in the park, adding a natural atmosphere to the scene. Multiple benches are placed in different locations in the park for people to sit and enjoy the surroundings."
},
{
"image": "Panda-Grassland.jpg",
"output": "The protagonist of the picture is a giant panda with black and white markings, standing on a lush, green grassland. This giant panda seems to be enjoying the warmth of the sun and the grass."
},
{
"image": "Bicycle-Flowers.jpg",
"output": "The image shows a green bicycle parked on a lane with yellow flowers, loaded with colorful blossoms. The bicycle is parked in front of a building, with two adjacent buildings forming a stark contrast.\nBesides the bicycle and flowers, there is a cup placed on the ground, possibly on or near the sidewalk."
},
{
"image": "dance.jpg",
"output": "In the image, a woman in a black dress is standing on a stage. She is performing on stage, surrounded by several people watching her, some of whom are holding mobile phones. These phones are scattered throughout the scene, possibly taken to capture the performance or communicate with the audience."
}
],
"vlm_stage": "sft"
},
{
"architecture": "qk_norm_muon",
"base_llm_stage": "pretrain",
"config": "muon_from_pretrain_pretrained",
"historical_command": "$ python eval_vlm.py --load 0 --model 0",
"output_count": 8,
"outputs": [
{
"image": "city-traffic.jpg",
"output": "A bus traveling from the city center to a tall building"
},
{
"image": "Astronaut-Space.jpg",
"output": "🤖️: A little boy on a spaceship"
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "A dog walking by the seaside"
},
{
"image": "Rainbow-Falls.jpg",
"output": "Water droplets falling from a cave"
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "A young man reading a book on an elderly person's bed."
},
{
"image": "Panda-Grassland.jpg",
"output": "A bear with a layer of cute black-and-white fur"
},
{
"image": "Bicycle-Flowers.jpg",
"output": "In a park in Paris, a little boy rides a bicycle along the riverbank."
},
{
"image": "dance.jpg",
"output": "Performers wearing colorful costumes and fashionable outfits during a performance."
}
],
"vlm_stage": "pretrained"
},
{
"architecture": "qk_norm_muon",
"base_llm_stage": "pretrain",
"config": "muon_from_pretrain_sft",
"historical_command": "$ python eval_vlm.py --load 0 --model 1",
"output_count": 8,
"outputs": [
{
"image": "city-traffic.jpg",
"output": "This image depicts a bustling city street at night, with high-rise buildings and heavy traffic. Cars are driving in different directions, creating a vibrant urban scene. Cars, trucks, and motorcycles flow continuously, showing the bustling activity on this street.\nA tall building stands prominently in the frame, forming part of the city skyline. The unique architectural style and scale of this building catch the attention of passersby, who might be drawn to appreciate its aesthetic value."
},
{
"image": "Astronaut-Space.jpg",
"output": "The image features a large spaceship or spacecraft, equipped with electronic devices and flight instruments. It appears to be from a museum or exhibition space, as various objects are displayed inside. These objects vary in size and seem to have been created by different professionals or scientists for visitors to explore."
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "In the picture, a woman sits on a beach bench holding a brown-and-white puppy. The puppy looks very happy, enjoying its time on the sand. In the background, two other people can be seen, also sitting on the same bench.\nBesides the woman and the puppy, various items appear in the scene, such as two handbags, a bench near the dog, a chair on the left, and a chair near the center of the scene."
},
{
"image": "Rainbow-Falls.jpg",
"output": "This image shows a stunning scene of rough seas extending to the horizon under a cloudy sky. A large mountain with a white dome stands in the center of the frame, becoming the focal point. The sea seems to be crashing against the horizon, creating an impression of strong waves.\nIn this scene, several people are scattered around, possibly enjoying the sea and the spectacular sky. Some of these people are closer to the coast, while others are farther away, likely admiring the breathtaking view."
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "An elderly man wearing glasses sits at a table in a library, surrounded by cluttered bookshelves and various books. He is sitting in a chair, focused on a book, possibly meditating or deep in thought. Behind him, a potted plant can be seen, adding a touch of greenery to the scene.\nThe library environment is very crowded, with bookshelves filled with all kinds of books, covering almost the entire wall. These books vary in size and position, some even overturned, giving the library a busy atmosphere."
},
{
"image": "Panda-Grassland.jpg",
"output": "In the picture, a large black-and-white bear sits on green grass, with a few flowers scattered around. It seems to be enjoying time outdoors, possibly in a grove or on the grassland."
},
{
"image": "Bicycle-Flowers.jpg",
"output": "The picture shows an outdoor garden with a bicycle parked between two other bicycles. Each bicycle has a basket filled with bright flowers. These flowers are arranged in various vases, adding color and vitality to the overall scene.\nIn the background, several people are in the garden, possibly enjoying a nice day. Some of them are near the bicycles, while others are elsewhere in the garden. Besides the bicycles, two cars are parked nearby, one near the foreground and the other farther away."
},
{
"image": "dance.jpg",
"output": "This image depicts a woman in elegant clothing walking down a wide walkway in the dark. She seems to be attending a dance, as she holds an umbrella, possibly to shield from rain or sunlight.\nChairs are placed along the walkway, likely for the audience to sit and enjoy the dance. Several kites can be seen in the picture, adding a joyful atmosphere to the dance."
}
],
"vlm_stage": "sft"
},
{
"architecture": "qk_norm_muon",
"base_llm_stage": "sft",
"config": "muon_from_sft_pretrained",
"historical_command": "$ python eval_vlm.py --load 0 --model 0",
"output_count": 8,
"outputs": [
{
"image": "city-traffic.jpg",
"output": "Monitoring of nighttime traffic lights"
},
{
"image": "Astronaut-Space.jpg",
"output": "A series of conversations between \"our ship\" and \"our ship.\""
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "A cute woman in a white long dress and a cute pet dog"
},
{
"image": "Rainbow-Falls.jpg",
"output": "From the mountaintop, water glistens in the sunlight."
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "An elderly person sitting on a bench in the park"
},
{
"image": "Panda-Grassland.jpg",
"output": "A cute zebra in the woods"
},
{
"image": "Bicycle-Flowers.jpg",
"output": "This video uses a horse working in a car."
},
{
"image": "dance.jpg",
"output": "A dancer performing on stage, wearing a stylish outfit."
}
],
"vlm_stage": "pretrained"
},
{
"architecture": "qk_norm_muon",
"base_llm_stage": "sft",
"config": "muon_from_sft_sft",
"historical_command": "$ python eval_vlm.py --load 0 --model 1",
"output_count": 8,
"outputs": [
{
"image": "city-traffic.jpg",
"output": "This photo captures a busy city street at night, with cars and trucks on the road. The street is lined with high-rise buildings, creating a bustling atmosphere. The street is crowded with traffic, including multiple cars and a truck. Some of these vehicles are stationary, while others are moving.\nA large number of high-rise buildings appear in the frame, highlighting the urban environment. These buildings vary in size and shape, forming the architectural landscape of the area. This nighttime photo emphasizes the dynamic nature of city life, with various elements combining to create the character of a night-time metropolis."
},
{
"image": "Astronaut-Space.jpg",
"output": "The scene in the image shows a soldier sitting on a blue ship, which contains a large TV screen. The soldier appears to be looking at a smartphone screen, possibly for monitoring or communicating with the outside world.\nThere are several other people on the ship, possibly crew members or scientists. Some of them are scattered around the ship, while others are near the TV screen. Additionally, two TV remote controls are placed on the ship, indicating they might be monitoring or interacting with the equipment."
},
{
"image": "Dog-Woman-Sea.jpg",
"output": "In the photo, a woman sits on the beach holding a large brown dog. She seems to be posing with the dog for the picture. The dog is on the left side of the frame.\nIn the background, multiple figures can be seen scattered across the beach. Some of them appear to be enjoying the beach, while others might be sightseeing or socializing."
},
{
"image": "Rainbow-Falls.jpg",
"output": "This image showcases an impressive fountain landscape, towering high on a grassy hill. The fountain occupies a significant portion of the frame, extending from the center to the far right, looking like a massive work of art.\nThe surroundings of the fountain, including the hill below and some trees in the background, provide a picturesque backdrop for this impressive fountain."
},
{
"image": "Chair-Elderly-Reading.jpg",
"output": "In the picture, an elderly man sits on a park bench reading a newspaper. He seems to be enjoying a peaceful time in the park, perhaps resting in the afternoon or pondering something.\nIn the background, multiple cars can be seen parked near the park, creating a tranquil atmosphere. Additionally, a person appears at the top of the frame, possibly watching the man read the newspaper, perhaps enjoying nature or waiting for someone."
},
{
"image": "Panda-Grassland.jpg",
"output": "The picture shows a small black-and-white panda standing on a patch of grass. This cute panda looks very relaxed and comfortable."
},
{
"image": "Bicycle-Flowers.jpg",
"output": "The picture shows a green bicycle parked next to a bench covered with a green tablecloth. This bench seems designed for gatherings, as it is near a potted plant.\nBesides the bicycle and the potted plant, a car is parked on the right side of the scene. A stop sign can be seen in the background, adding an urban touch to the image."
},
{
"image": "dance.jpg",
"output": "The image shows a woman in a tuxedo standing on a stage, holding a microphone, seemingly delivering an engaging performance. She looks like an actor or performer, with several people around her, possibly the audience or participants.\nVarious items are placed on the stage, including a wine glass and several bottles. Some of these bottles are near the stage, while others are scattered in the background. The scene captures an event on stage, with the actor or performer delivering a memorable show for the audience."
}
],
"vlm_stage": "sft"
}
],
"configs": [
"without_muon_pretrained",
"without_muon_sft",
"muon_from_dpo_pretrained",
"muon_from_dpo_sft",
"muon_from_pretrain_pretrained",
"muon_from_pretrain_sft",
"muon_from_sft_pretrained",
"muon_from_sft_sft"
],
"experiment": "8-4",
"images": [
"Rainbow-Falls.jpg",
"Dog-Woman-Sea.jpg",
"dance.jpg",
"Astronaut-Space.jpg",
"city-traffic.jpg",
"Panda-Grassland.jpg",
"Bicycle-Flowers.jpg",
"Chair-Elderly-Reading.jpg"
],
"output_count": 64,
"schema_version": "exp8-4-retained-outputs-v1",
"source_report": "chapter8/MiniMind-pretrain/README.md",
"source_report_sha256": "a8e1df1a9ee5cf013995e9ff3b963621485a838c97456b9f30e65ea9fdf55d50"
}
chapter8/MiniMind-pretrain/validation/runs/exp8-4-training-report-20260731-v1/summary.json
+858
View File
@@ -0,0 +1,858 @@
{
"acceptance": {
"all_64_historical_outputs_retained": true,
"all_eight_configuration_cells_retained": true,
"checkpoints_not_an_acceptance_artifact": true,
"eight_image_aware_arm_blind_judgments": true,
"future_reproduction_commands_declared": true,
"historical_provenance_limitations_explicit": true,
"historical_report_content_hashed": true,
"immutable_dataset_clip_and_eval_image_inputs_frozen": true,
"immutable_original_and_improved_source_revisions_frozen": true,
"passed": true,
"raw_judge_requests_responses_ids_usage_latency_retained": true,
"request_images_match_pinned_sha256": true,
"same_eight_images_present_in_every_cell": true
},
"best_counts": {
"muon_from_dpo_pretrained": 0,
"muon_from_dpo_sft": 2,
"muon_from_pretrain_pretrained": 1,
"muon_from_pretrain_sft": 0,
"muon_from_sft_pretrained": 2,
"muon_from_sft_sft": 1,
"without_muon_pretrained": 1,
"without_muon_sft": 1
},
"config_averages": {
"muon_from_dpo_pretrained": {
"coverage": 0.875,
"grounding_accuracy": 1.0,
"hallucination_control": 1.5,
"overall": 0.9375,
"visual_specificity": 0.375
},
"muon_from_dpo_sft": {
"coverage": 1.875,
"grounding_accuracy": 1.375,
"hallucination_control": 1.125,
"overall": 1.5312,
"visual_specificity": 1.75
},
"muon_from_pretrain_pretrained": {
"coverage": 0.75,
"grounding_accuracy": 0.75,
"hallucination_control": 1.375,
"overall": 0.875,
"visual_specificity": 0.625
},
"muon_from_pretrain_sft": {
"coverage": 1.375,
"grounding_accuracy": 1.125,
"hallucination_control": 0.5,
"overall": 1.0312,
"visual_specificity": 1.125
},
"muon_from_sft_pretrained": {
"coverage": 1.375,
"grounding_accuracy": 1.875,
"hallucination_control": 2.0,
"overall": 1.5312,
"visual_specificity": 0.875
},
"muon_from_sft_sft": {
"coverage": 1.5,
"grounding_accuracy": 1.25,
"hallucination_control": 1.125,
"overall": 1.2812,
"visual_specificity": 1.25
},
"without_muon_pretrained": {
"coverage": 1.5,
"grounding_accuracy": 1.875,
"hallucination_control": 2.625,
"overall": 1.7188,
"visual_specificity": 0.875
},
"without_muon_sft": {
"coverage": 2.375,
"grounding_accuracy": 2.125,
"hallucination_control": 1.375,
"overall": 1.9062,
"visual_specificity": 1.75
}
},
"experiment": "8-4",
"isolated_original_vs_qk_norm_muon_from_sft": {
"pretrained": {
"delta": -0.1876,
"original": 1.7188,
"qk_norm_muon_from_sft": 1.5312
},
"sft": {
"delta": -0.625,
"original": 1.9062,
"qk_norm_muon_from_sft": 1.2812
}
},
"judge": {
"blind_seed": 740731,
"calls": 8,
"image_aware": true,
"model": "doubao-seed-1-6-250615",
"provider": "ark",
"response_ids": [
"021785497883895355cac89e6d983ae8d30678a5b54f9afc7a30b",
"0217854978839024aa8fafbe9055982037cfb2f5ec29c229036d1",
"021785497883901cbe31712b8a2594f4e458beea699b0faabecc8",
"0217854978838946c88a39b89e3e3930a4a1be3bade8084753bc5",
"0217854979402514aa8fafbe9055982037cfb2f5ec29c22876438",
"021785497944549bdf0d8876ec55e469c432250c926544c9a9a49",
"0217854979562513b2c90331db17702ff1ef9ef62383e60dd8c12",
"02178549797786577f9606cef83fa80b3d313b2e208ca1cdfc167"
],
"total_latency_ms": 557409.335,
"total_tokens": 43094
},
"limitations": [
"Historical base-LLM and VLM checkpoints are intentionally not distributed and were not recreated in this audit.",
"Historical source revisions, dataset identities, RNG state, hardware image, and stepwise logs were not retained.",
"Current immutable pins define a future reproduction contract and are not represented as the exact historical run.",
"The English captions are translations in a bilingual report, so translation can affect judging.",
"One image-aware judge call evaluates all eight anonymous candidates per image; scores are descriptive, not a powered significance test.",
"QK-Norm and Muon change together in the improved arm, so the report does not attribute effects to Muon alone."
],
"per_image_config_scores": {
"Astronaut-Space.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 2,
"grounding_accuracy": 2,
"hallucination_control": 1,
"material_errors": [
"'sent to a new Earth' is not visible; no indication of a mission to a new Earth"
],
"rationale": "Mentions astronaut ('spaceman') but includes unsupported claim about being 'sent to a new Earth'.",
"visual_specificity": 1
},
"muon_from_dpo_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"spacesuit is white, not black; no glasses; astronaut is standing, not sitting; no boats or plane in image; astronaut is on the left, not far right"
],
"rationale": "Contains multiple incorrect details (color, position, invented objects) and misidentifies actions.",
"visual_specificity": 0
},
"muon_from_pretrain_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"'little boy' is not present; subject is an adult astronaut"
],
"rationale": "Incorrectly identifies subject as a little boy instead of an astronaut.",
"visual_specificity": 0
},
"muon_from_pretrain_sft": {
"coverage": 2,
"grounding_accuracy": 2,
"hallucination_control": 1,
"material_errors": [
"'museum or exhibition space' is incorrect; scene is inside a functional spacecraft, not a museum; 'objects displayed for visitors' is not visible"
],
"rationale": "Partially mentions spacecraft and instruments but incorrectly claims it's a museum exhibit.",
"visual_specificity": 2
},
"muon_from_sft_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no conversations or dialogue visible; irrelevant to image content"
],
"rationale": "No visual support for conversations; unrelated to the scene.",
"visual_specificity": 0
},
"muon_from_sft_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"subject is an astronaut, not a soldier; no other people present; no smartphone or TV remote controls visible; not a 'blue ship' but a spacecraft interior"
],
"rationale": "Completely misrepresents the scene with invented elements.",
"visual_specificity": 0
},
"without_muon_pretrained": {
"coverage": 2,
"grounding_accuracy": 2,
"hallucination_control": 3,
"material_errors": [
"astronaut is inside a spacecraft, not performing a spacewalk (no spacewalk visible)"
],
"rationale": "Correctly identifies astronaut but incorrectly claims a spacewalk; astronaut is inside the spacecraft.",
"visual_specificity": 2
},
"without_muon_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no commercial airplane; no small hill or barn; no people observing; central object is a space station, not an airplane"
],
"rationale": "Describes an invented scene with no relation to the actual image.",
"visual_specificity": 0
}
},
"Bicycle-Flowers.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"riding a bicycle (bicycle is parked)",
"bicycles everywhere (only one bicycle)",
"many cars (no cars)",
"this is my bicycle (no ownership indicated)"
],
"rationale": "Contains entirely false claims about riding, multiple bicycles, cars, and personal ownership; no accurate scene elements.",
"visual_specificity": 0
},
"muon_from_dpo_sft": {
"coverage": 4,
"grounding_accuracy": 4,
"hallucination_control": 3,
"material_errors": [
"two adjacent buildings (only one building visible)",
"cup placed on the ground (no cup)"
],
"rationale": "Accurately describes green parked bicycle with yellow/colorful flowers in front of a building; minor errors with extra buildings and cup.",
"visual_specificity": 4
},
"muon_from_pretrain_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"park in Paris (no park/Paris context)",
"little boy rides (no boy; bicycle is parked)",
"riverbank (no riverbank)"
],
"rationale": "Completely invented scenario with no relation to the image's bicycle, flowers, or building.",
"visual_specificity": 0
},
"muon_from_pretrain_sft": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 0,
"material_errors": [
"outdoor garden (no garden; sidewalk next to building)",
"two other bicycles (only one bicycle)",
"vases (flowers in baskets, not vases)",
"several people (no people)",
"two cars (no cars)"
],
"rationale": "Incorrectly claims garden setting, multiple bicycles, people, cars, and vases; only bicycle with flower baskets partially accurate.",
"visual_specificity": 1
},
"muon_from_sft_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"video (still image)",
"horse working in a car (no horse or car)"
],
"rationale": "Entirely unrelated to the image; mentions horse and car in a video, neither present.",
"visual_specificity": 0
},
"muon_from_sft_sft": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 0,
"material_errors": [
"bench with green tablecloth (no bench)",
"potted plant (no potted plant)",
"car parked on right (no car)",
"stop sign (no stop sign)"
],
"rationale": "Invents bench, potted plant, car, and stop sign; only green bicycle color is accurate but irrelevant to main scene.",
"visual_specificity": 1
},
"without_muon_pretrained": {
"coverage": 3,
"grounding_accuracy": 3,
"hallucination_control": 4,
"material_errors": [],
"rationale": "Accurately references the bicycle, tires/wheels, and flowers, though lacks details like green color and baskets.",
"visual_specificity": 2
},
"without_muon_sft": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 0,
"material_errors": [
"row of vases (no vases; flowers in baskets)",
"three potted plants (no potted plants)",
"two people (no people)"
],
"rationale": "Mentions bicycle and building but invents vases, potted plants, and people not present in the image.",
"visual_specificity": 1
}
},
"Chair-Elderly-Reading.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 3,
"grounding_accuracy": 5,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Correctly identifies elderly person on park bench with no hallucinations, but lacks reading activity.",
"visual_specificity": 2
},
"muon_from_dpo_sft": {
"coverage": 4,
"grounding_accuracy": 4,
"hallucination_control": 4,
"material_errors": [
"multiple benches not visible"
],
"rationale": "Correctly identifies elderly man on park bench holding a book, trees in background; only error is multiple benches.",
"visual_specificity": 4
},
"muon_from_pretrain_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"young man instead of elderly",
"bed instead of park bench"
],
"rationale": "No young man or bed visible; subject and setting are entirely incorrect.",
"visual_specificity": 0
},
"muon_from_pretrain_sft": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 0,
"material_errors": [
"library instead of park",
"table instead of bench",
"cluttered bookshelves not present",
"potted plant not visible"
],
"rationale": "Incorrect library setting; no table, bookshelves, or potted plant; only elderly man with glasses is correct.",
"visual_specificity": 1
},
"muon_from_sft_pretrained": {
"coverage": 3,
"grounding_accuracy": 5,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Correctly identifies elderly person on park bench with no hallucinations, but lacks reading activity.",
"visual_specificity": 1
},
"muon_from_sft_sft": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 0,
"material_errors": [
"reading newspaper instead of book",
"multiple cars not visible",
"person at top of frame not present"
],
"rationale": "Reads book (not newspaper); no cars or person at top; bench and park are correct but other details invented.",
"visual_specificity": 1
},
"without_muon_pretrained": {
"coverage": 2,
"grounding_accuracy": 2,
"hallucination_control": 4,
"material_errors": [
"incorrect gender (woman instead of man)"
],
"rationale": "Correct park setting and reading a book, but misidentifies gender as woman.",
"visual_specificity": 2
},
"without_muon_sft": {
"coverage": 3,
"grounding_accuracy": 3,
"hallucination_control": 2,
"material_errors": [
"several cars parked nearby not visible",
"another bench in the background not present"
],
"rationale": "Correct elderly man with glasses reading a book on park bench, but invents cars and another bench.",
"visual_specificity": 3
}
},
"Dog-Woman-Sea.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no ball, palm, or throwing action in image"
],
"rationale": "Completely irrelevant; no ball or throwing present.",
"visual_specificity": 0
},
"muon_from_dpo_sft": {
"coverage": 1,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"woman sits on sand, not a bench",
"woman not holding dog; dog sits beside her",
"woman wears sleeveless top/jeans, not a dress",
"no other people in background"
],
"rationale": "Contains multiple hallucinations: bench, held dog, dress, and other people, all absent.",
"visual_specificity": 0
},
"muon_from_pretrain_pretrained": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 4,
"material_errors": [
"dog is sitting, not walking",
"missing woman sitting beside dog"
],
"rationale": "Correct about dog and seaside but misses woman and incorrectly states dog is walking.",
"visual_specificity": 1
},
"muon_from_pretrain_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no beach bench; woman sits on sand",
"woman not holding dog; dog sits beside her",
"dog is adult, not a puppy",
"no other people, handbags, chairs, or bench"
],
"rationale": "Filled with hallucinated objects (bench, handbags, chairs) and incorrect actions (holding puppy).",
"visual_specificity": 0
},
"muon_from_sft_pretrained": {
"coverage": 2,
"grounding_accuracy": 2,
"hallucination_control": 3,
"material_errors": [
"woman wears light blue sleeveless top and blue jeans, not white long dress"
],
"rationale": "Identifies woman and dog but misrepresents clothing (white long dress vs. blue top/jeans).",
"visual_specificity": 1
},
"muon_from_sft_sft": {
"coverage": 2,
"grounding_accuracy": 1,
"hallucination_control": 1,
"material_errors": [
"woman not holding dog; dog sits beside her",
"no multiple figures in background",
"dog is brown and white, not just brown"
],
"rationale": "Incorrectly claims woman holds dog and background has multiple people; dog color misrepresented.",
"visual_specificity": 1
},
"without_muon_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no young person decorating beach",
"no dog's head; whole dog sits beside woman"
],
"rationale": "Entirely misrepresents scene; no decorating or dog's head present.",
"visual_specificity": 0
},
"without_muon_sft": {
"coverage": 3,
"grounding_accuracy": 4,
"hallucination_control": 4,
"material_errors": [
"dog is not on a blue and white checkered blanket; dog sits on sand"
],
"rationale": "Correctly identifies woman sitting on beach with dog beside her; falsely claims dog sits on a checkered blanket (not present).",
"visual_specificity": 3
}
},
"Panda-Grassland.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"subject is a black-and-white panda, not a white bear"
],
"rationale": "Refers to the panda as a 'white bear'; panda has distinct black-and-white fur.",
"visual_specificity": 0
},
"muon_from_dpo_sft": {
"coverage": 4,
"grounding_accuracy": 3,
"hallucination_control": 2,
"material_errors": [
"panda is lying, not standing; no evidence of sun warmth"
],
"rationale": "Accurately identifies giant panda with black and white markings on lush green grassland but incorrectly states it is standing and mentions unevidenced sun warmth.",
"visual_specificity": 4
},
"muon_from_pretrain_pretrained": {
"coverage": 3,
"grounding_accuracy": 4,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Accurately describes the panda as a bear with cute black-and-white fur, with no incorrect details.",
"visual_specificity": 3
},
"muon_from_pretrain_sft": {
"coverage": 3,
"grounding_accuracy": 2,
"hallucination_control": 1,
"material_errors": [
"panda is lying, not sitting; no flowers visible; no grove (only grass)"
],
"rationale": "Identifies black-and-white bear on green grass but includes hallucinations (flowers, grove) and incorrect position (sitting).",
"visual_specificity": 2
},
"muon_from_sft_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"subject is panda, not zebra; setting is grass, not woods"
],
"rationale": "Incorrectly identifies subject as zebra and setting as woods; image shows a panda on grass.",
"visual_specificity": 0
},
"muon_from_sft_sft": {
"coverage": 3,
"grounding_accuracy": 2,
"hallucination_control": 3,
"material_errors": [
"panda is lying, not standing"
],
"rationale": "Identifies black-and-white panda and grass but incorrectly states the panda is standing (it is lying).",
"visual_specificity": 2
},
"without_muon_pretrained": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 0,
"material_errors": [
"no evidence of zoo; panda is not eating bamboo (lying on grass)"
],
"rationale": "States panda is in a zoo eating bamboo; image shows panda lying on grass with no bamboo or zoo elements.",
"visual_specificity": 0
},
"without_muon_sft": {
"coverage": 3,
"grounding_accuracy": 2,
"hallucination_control": 1,
"material_errors": [
"panda is not wearing glasses; no evidence of sun or rain; conflicting positions (sitting vs lying)"
],
"rationale": "Identifies black-and-white panda on grass and mentions lying/resting but includes hallucinations (glasses, sun/rain) and conflicting position (sitting).",
"visual_specificity": 1
}
},
"Rainbow-Falls.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Vaguely refers to a 'water landscape' but lacks specific details about the waterfall or rainbow.",
"visual_specificity": 0
},
"muon_from_dpo_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"fountain is not present (it is a waterfall)",
"umbrella on water is not present",
"visitors are not present"
],
"rationale": "Incorrectly identifies the waterfall as a 'fountain' and adds non-existent elements like umbrella and visitors.",
"visual_specificity": 0
},
"muon_from_pretrain_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"cave is not present",
"water source is a waterfall, not droplets from a cave"
],
"rationale": "Invents a 'cave' and misrepresents the water source; does not depict the waterfall.",
"visual_specificity": 0
},
"muon_from_pretrain_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"rough seas not present",
"cloudy sky not present (sky is clear)",
"mountain with white dome not present",
"people not present"
],
"rationale": "Describes an unrelated seascape with people and mountains, not the waterfall scene.",
"visual_specificity": 0
},
"muon_from_sft_pretrained": {
"coverage": 2,
"grounding_accuracy": 3,
"hallucination_control": 3,
"material_errors": [
"missing mention of waterfall and rainbow",
"perspective as 'from the mountaintop' is unclear"
],
"rationale": "Accurately notes water glistening in sunlight but omits key elements (waterfall, rainbow) and has unclear perspective.",
"visual_specificity": 2
},
"muon_from_sft_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"waterfall is misidentified as a fountain"
],
"rationale": "Falsely labels the waterfall as a 'fountain landscape' with no basis in the image.",
"visual_specificity": 1
},
"without_muon_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"rainbow bridge is not present",
"focus on water droplets is incorrect"
],
"rationale": "Mentions non-existent 'rainbow bridge' and misfocuses on droplets; fails to describe the waterfall.",
"visual_specificity": 0
},
"without_muon_sft": {
"coverage": 3,
"grounding_accuracy": 2,
"hallucination_control": 1,
"material_errors": [
"highway is not present",
"giant rainbow flag is not present"
],
"rationale": "Correctly identifies a large waterfall but includes hallucinated elements (highway, rainbow flag) not in the image.",
"visual_specificity": 2
}
},
"city-traffic.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 1,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no elevators visible; main subject (heavy traffic) not mentioned"
],
"rationale": "Irrelevant focus on elevators and sidewalks; misses the prominent heavy traffic and tall buildings.",
"visual_specificity": 0
},
"muon_from_dpo_sft": {
"coverage": 2,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no parked cars (all cars are in traffic); no traffic lights visible; no bus present"
],
"rationale": "Falsely claims parked cars, traffic lights, and a bus; these elements are not visible in the image.",
"visual_specificity": 2
},
"muon_from_pretrain_pretrained": {
"coverage": 1,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"no bus visible in the image"
],
"rationale": "A tall building is present, but there is no bus. The main subject is heavy traffic, not a bus traveling to a building.",
"visual_specificity": 1
},
"muon_from_pretrain_sft": {
"coverage": 4,
"grounding_accuracy": 3,
"hallucination_control": 2,
"material_errors": [
"no motorcycles visible; no passersby (pedestrians) visible"
],
"rationale": "Correctly identifies busy street, traffic, tall buildings, but falsely includes motorcycles and passersby.",
"visual_specificity": 3
},
"muon_from_sft_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"mentions 'monitoring' which is not visible; traffic lights are not the main subject and not clearly present"
],
"rationale": "The image depicts a busy nighttime city street with traffic and buildings, not monitoring of traffic lights. No monitoring activity or distinct traffic lights are visible.",
"visual_specificity": 0
},
"muon_from_sft_sft": {
"coverage": 5,
"grounding_accuracy": 5,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Accurately describes the nighttime busy city street with cars, trucks, high-rise buildings, and bustling traffic (stationary and moving) without hallucinations.",
"visual_specificity": 4
},
"without_muon_pretrained": {
"coverage": 1,
"grounding_accuracy": 2,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Generic mention of 'city streets at nightfall' is accurate but lacks specific details about traffic, buildings, or activity.",
"visual_specificity": 0
},
"without_muon_sft": {
"coverage": 4,
"grounding_accuracy": 3,
"hallucination_control": 2,
"material_errors": [
"no pedestrians visible; no traffic lights clearly visible"
],
"rationale": "Correctly identifies busy street, traffic, cars, truck, tall buildings, and streetlights, but falsely mentions pedestrians and traffic lights which are not present.",
"visual_specificity": 3
}
},
"dance.jpg": {
"muon_from_dpo_pretrained": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 1,
"material_errors": [
"actors (image shows a dancer, not actors)"
],
"rationale": "Incorrectly identifies the subject as 'actors' instead of a dancer; no other accurate details.",
"visual_specificity": 0
},
"muon_from_dpo_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"black dress (actual: light-colored dress)",
"several people watching (no people present)",
"mobile phones (no phones in image)"
],
"rationale": "Incorrectly describes the dress as black (it is light-colored) and invents non-existent people and mobile phones.",
"visual_specificity": 0
},
"muon_from_pretrain_pretrained": {
"coverage": 1,
"grounding_accuracy": 1,
"hallucination_control": 2,
"material_errors": [
"performers (only one dancer)",
"colorful costumes (dress is light-colored, not colorful)"
],
"rationale": "Vaguely mentions a performance but incorrectly refers to multiple 'performers' and 'colorful costumes' (image has one dancer in a light dress).",
"visual_specificity": 0
},
"muon_from_pretrain_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"walking down walkway (dancing on stage)",
"holding umbrella (no umbrella)",
"chairs along walkway (no chairs)",
"kites (no kites)"
],
"rationale": "Contains multiple hallucinations: walkway, umbrella, chairs, and kites are all absent; the subject is dancing on stage, not walking.",
"visual_specificity": 0
},
"muon_from_sft_pretrained": {
"coverage": 4,
"grounding_accuracy": 5,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Accurately identifies a dancer performing on stage in a stylish outfit, with no false claims; aligns with visible elements (single dancer, stage, elegant dress).",
"visual_specificity": 3
},
"muon_from_sft_sft": {
"coverage": 0,
"grounding_accuracy": 0,
"hallucination_control": 0,
"material_errors": [
"tuxedo (wearing a dress)",
"holding microphone (no microphone)",
"several people around (no people)",
"wine glass and bottles (no such items)"
],
"rationale": "Contains multiple false claims: tuxedo (dress), microphone, people, and wine glasses/bottles are all absent.",
"visual_specificity": 0
},
"without_muon_pretrained": {
"coverage": 3,
"grounding_accuracy": 5,
"hallucination_control": 5,
"material_errors": [],
"rationale": "Correctly states a dancer is performing on stage but lacks visual details (e.g., outfit description).",
"visual_specificity": 1
},
"without_muon_sft": {
"coverage": 2,
"grounding_accuracy": 2,
"hallucination_control": 1,
"material_errors": [
"dance steps soaring high (dancer is on stage floor)",
"several chairs on stage (no chairs)",
"clock on stage (no clock)"
],
"rationale": "Mentions a dance on stage but invents 'soaring steps' (dancer is grounded) and non-existent chairs/clock.",
"visual_specificity": 1
}
}
},
"ranking_by_overall": [
"without_muon_sft",
"without_muon_pretrained",
"muon_from_dpo_sft",
"muon_from_sft_pretrained",
"muon_from_sft_sft",
"muon_from_pretrain_sft",
"muon_from_dpo_pretrained",
"muon_from_pretrain_pretrained"
],
"retained": {
"cells": 8,
"images": 8,
"outputs": 64
},
"schema_version": "exp8-4-summary-v1",
"scientific_findings": {
"author_claims_are_historical_observations_not_acceptance_gates": true,
"muon_only_causal_claim_avoided": true,
"sft_minus_pretrained_average": 0.1718,
"top_configuration": "without_muon_sft",
"top_configuration_overall": 1.9062
},
"stage_averages": {
"pretrained": {
"coverage": 1.125,
"grounding_accuracy": 1.375,
"hallucination_control": 1.875,
"overall": 1.2656,
"visual_specificity": 0.6875
},
"sft": {
"coverage": 1.7812,
"grounding_accuracy": 1.4688,
"hallucination_control": 1.0312,
"overall": 1.4374,
"visual_specificity": 1.4688
}
},
"status": "passed"
}
chapter8/MiniMind-pretrain/validation/test_training_report_audit.py
+135
View File
@@ -0,0 +1,135 @@
from __future__ import annotations
import hashlib
import json
import shutil
from pathlib import Path
import pytest
import run_training_report_audit as audit
import validate_evidence as validator
def digest(path: Path) -> str:
return hashlib.sha256(path.read_bytes()).hexdigest()
def write_json(path: Path, value: object) -> None:
path.write_text(
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n",
encoding="utf-8",
)
def test_historical_report_parser_retains_complete_six_cell_matrix() -> None:
retained = audit.parse_retained_outputs()
assert retained["cell_count"] == 6
assert retained["output_count"] == 49
assert {
(cell["arm"], cell["stage"]): cell["pair_count"]
for cell in retained["cells"]
} == audit.EXPECTED_COUNTS
comparisons = audit.selected_comparisons(retained)
assert len(comparisons) == 8
assert {row["stage"] for row in comparisons} == set(audit.STAGES)
assert all(set(row["arms"]) == set(audit.ARMS) for row in comparisons)
def test_judge_requests_are_arm_blind_and_bound_to_exact_outputs() -> None:
retained = audit.parse_retained_outputs()
for comparison in audit.selected_comparisons(retained):
mapping = audit.blind_mapping(comparison["case_id"])
payload = audit.judge_payload(comparison, mapping, "judge-model")
serialized = json.dumps(payload, ensure_ascii=False).lower()
assert "qk_norm_muon" not in serialized
assert '"original"' not in serialized
user_payload = json.loads(payload["messages"][1]["content"])
for label, arm in mapping.items():
assert user_payload["candidates"][label]["historical_output"] == comparison["arms"][arm]["output"]
def make_validation_copy(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> tuple[Path, Path]:
latest = json.loads(audit.LATEST_PATH.read_text(encoding="utf-8"))
canonical_run = audit.EXPERIMENT_DIR / latest["run_dir"]
temp_repo = tmp_path / "repo"
temp_experiment = temp_repo / "chapter8/MiniMind-pretrain"
temp_run = temp_experiment / latest["run_dir"]
temp_run.parent.mkdir(parents=True)
shutil.copytree(canonical_run, temp_run)
manifest = json.loads((canonical_run / "manifest.json").read_text(encoding="utf-8"))
for record in manifest["inputs"]:
source = audit.REPO_ROOT / record["path"]
destination = temp_repo / record["path"]
destination.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(source, destination)
temp_latest = temp_experiment / "validation/latest.json"
temp_latest.parent.mkdir(parents=True, exist_ok=True)
shutil.copy2(audit.LATEST_PATH, temp_latest)
monkeypatch.setattr(validator, "REPO_ROOT", temp_repo)
monkeypatch.setattr(validator, "EXPERIMENT_DIR", temp_experiment)
return temp_latest, temp_run
def refresh_outer_hashes(latest_path: Path, run_dir: Path, artifact_name: str) -> None:
manifest_path = run_dir / "manifest.json"
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
artifact_path = run_dir / artifact_name
record = next(record for record in manifest["artifacts"] if record["path"] == artifact_name)
record["bytes"] = artifact_path.stat().st_size
record["sha256"] = digest(artifact_path)
write_json(manifest_path, manifest)
latest = json.loads(latest_path.read_text(encoding="utf-8"))
latest["manifest_sha256"] = digest(manifest_path)
write_json(latest_path, latest)
def test_canonical_evidence_passes_fail_closed_validator() -> None:
result = validator.validate()
assert result["status"] == "passed"
assert result["outputs_verified"] == 49
assert result["judge_receipts_verified"] == 8
def test_validator_rejects_raw_response_normalization_tamper(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
latest_path, run_dir = make_validation_copy(tmp_path, monkeypatch)
receipts_path = run_dir / "judge_receipts.json"
receipts = json.loads(receipts_path.read_text(encoding="utf-8"))
response = receipts["calls"][0]["response"]
raw_judgment = json.loads(response["choices"][0]["message"]["content"])
raw_judgment["winner"] = "tie" if raw_judgment["winner"] != "tie" else "A"
response["choices"][0]["message"]["content"] = json.dumps(raw_judgment)
write_json(receipts_path, receipts)
refresh_outer_hashes(latest_path, run_dir, "judge_receipts.json")
with pytest.raises(AssertionError, match="normalized judgment"):
validator.validate(latest_path)
def test_validator_rejects_retained_output_request_binding_tamper(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
latest_path, run_dir = make_validation_copy(tmp_path, monkeypatch)
retained_path = run_dir / "retained_outputs.json"
retained = json.loads(retained_path.read_text(encoding="utf-8"))
retained["cells"][0]["pairs"][3]["output"] += " altered"
write_json(retained_path, retained)
refresh_outer_hashes(latest_path, run_dir, "retained_outputs.json")
with pytest.raises(AssertionError, match="not bound to the retained output"):
validator.validate(latest_path)
def test_validator_rejects_frozen_dataset_revision_tamper(
tmp_path: Path, monkeypatch: pytest.MonkeyPatch
) -> None:
latest_path, run_dir = make_validation_copy(tmp_path, monkeypatch)
contract_path = run_dir / "reproduction_contract.json"
contract = json.loads(contract_path.read_text(encoding="utf-8"))
contract["future_reproduction"]["dataset"]["revision"] = "0" * 40
write_json(contract_path, contract)
refresh_outer_hashes(latest_path, run_dir, "reproduction_contract.json")
with pytest.raises(AssertionError, match="dataset revision mismatch"):
validator.validate(latest_path)
chapter8/MiniMind-pretrain/validation/test_vlm_training_report_audit.py
+161
View File
@@ -0,0 +1,161 @@
import hashlib
import json
import shutil
import sys
from pathlib import Path
import pytest
HERE = Path(__file__).resolve().parent
sys.path.insert(0, str(HERE))
import run_vlm_training_report_audit as audit
import validate_vlm_evidence as validator
RUN_DIR = HERE / "runs" / audit.DEFAULT_RUN_ID
def write_json(path: Path, value):
path.write_text(
json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n", encoding="utf-8"
)
def reseal_artifact(run_dir: Path, name: str):
manifest_path = run_dir / "manifest.json"
manifest = json.loads(manifest_path.read_text(encoding="utf-8"))
artifact = next(row for row in manifest["artifacts"] if row["path"] == name)
artifact["bytes"] = (run_dir / name).stat().st_size
artifact["sha256"] = hashlib.sha256((run_dir / name).read_bytes()).hexdigest()
write_json(manifest_path, manifest)
def copied_run(tmp_path: Path) -> Path:
target = tmp_path / "run"
shutil.copytree(RUN_DIR, target)
return target
def test_parser_retains_all_eight_cells_and_64_outputs():
retained = audit.parse_retained_outputs()
assert retained["cell_count"] == 8
assert retained["output_count"] == 64
assert tuple(cell["config"] for cell in retained["cells"]) == audit.CONFIGS
assert all(cell["output_count"] == 8 for cell in retained["cells"])
assert all(
{row["image"] for row in cell["outputs"]} == set(audit.IMAGE_FILES)
for cell in retained["cells"]
)
def test_blind_mapping_is_deterministic_bijective_and_image_specific():
mappings = [audit.blind_mapping(image) for image in audit.IMAGE_FILES]
assert all(set(mapping) == set(audit.LABELS) for mapping in mappings)
assert all(set(mapping.values()) == set(audit.CONFIGS) for mapping in mappings)
assert all(
mapping == audit.blind_mapping(image)
for image, mapping in zip(audit.IMAGE_FILES, mappings, strict=True)
)
assert len({tuple(mapping.items()) for mapping in mappings}) > 1
def test_reproduction_contract_separates_original_and_improved_sources():
contract = audit.reproduction_contract()
vlm = contract["future_reproduction"]["vlm_source"]
commands = contract["future_reproduction"]["commands"]
assert vlm["original_revision"] == audit.ORIGINAL_VLM_REVISION
assert vlm["qk_norm_muon_revision"] == audit.IMPROVED_VLM_REVISION
assert vlm["original_revision"] != vlm["qk_norm_muon_revision"]
assert audit.ORIGINAL_VLM_REVISION in commands["original_source"]
assert audit.IMPROVED_VLM_REVISION in commands["improved_source"]
assert "checkout --detach" in commands["original_source"]
assert "checkout --detach" in commands["improved_source"]
assert (
contract["historical_evidence_boundary"]["historical_vlm_checkpoint_hashes_retained"]
is False
)
assert contract["checkpoint_policy"]["acceptance_artifact"] is False
def test_canonical_evidence_passes_fail_closed_validator():
result = validator.validate_run(RUN_DIR)
assert result["status"] == "passed"
assert result["cells"] == 8
assert result["outputs"] == 64
assert result["judge_receipts"] == 8
def test_vlm_latest_pointer_does_not_overwrite_experiment_8_3():
assert audit.LATEST_PATH.name == "latest_vlm.json"
vlm_latest = json.loads(audit.LATEST_PATH.read_text(encoding="utf-8"))
llm_latest = json.loads((HERE / "latest.json").read_text(encoding="utf-8"))
assert vlm_latest["experiment"] == "8-4"
assert llm_latest["experiment"] == "8-3"
def test_receipts_are_real_image_aware_unique_and_arm_blind():
receipts = json.loads((RUN_DIR / "judge_receipts.json").read_text(encoding="utf-8"))["calls"]
assert len({row["response_id"] for row in receipts}) == 8
retained = audit.parse_retained_outputs()
for receipt in receipts:
assert receipt["http_status"] == 200
assert receipt["usage"]["total_tokens"] > 0
content = receipt["request"]["messages"][1]["content"]
assert content[0]["type"] == "image_url"
assert content[0]["image_url"]["url"].startswith("data:image/jpeg;base64,")
prompt = json.loads(content[1]["text"])
expected = audit.outputs_for_image(retained, receipt["image"])
assert prompt["candidates"] == {
label: expected[config] for label, config in receipt["blind_map"].items()
}
assert not any(config in content[1]["text"] for config in audit.CONFIGS)
def test_tampered_retained_output_fails_even_after_resealing(tmp_path):
run_dir = copied_run(tmp_path)
path = run_dir / "retained_outputs.json"
data = json.loads(path.read_text(encoding="utf-8"))
data["cells"][0]["outputs"][0]["output"] += " tampered"
write_json(path, data)
reseal_artifact(run_dir, path.name)
with pytest.raises(validator.EvidenceError, match="retained outputs"):
validator.validate_run(run_dir, verify_latest=False)
def test_tampered_request_image_fails_even_after_resealing(tmp_path):
run_dir = copied_run(tmp_path)
path = run_dir / "judge_receipts.json"
data = json.loads(path.read_text(encoding="utf-8"))
url = data["calls"][0]["request"]["messages"][1]["content"][0]["image_url"]["url"]
prefix, encoded = url.split(",", 1)
replacement = "A" if encoded[-2] != "A" else "B"
data["calls"][0]["request"]["messages"][1]["content"][0]["image_url"]["url"] = (
prefix + "," + encoded[:-2] + replacement + encoded[-1]
)
write_json(path, data)
reseal_artifact(run_dir, path.name)
with pytest.raises(validator.EvidenceError, match="request image bytes"):
validator.validate_run(run_dir, verify_latest=False)
def test_normalized_judgment_must_match_raw_provider_response(tmp_path):
run_dir = copied_run(tmp_path)
path = run_dir / "judge_receipts.json"
data = json.loads(path.read_text(encoding="utf-8"))
current = data["calls"][0]["judgment"]["candidates"]["A"]["grounding_accuracy"]
data["calls"][0]["judgment"]["candidates"]["A"]["grounding_accuracy"] = 0 if current != 0 else 1
write_json(path, data)
reseal_artifact(run_dir, path.name)
with pytest.raises(validator.EvidenceError, match="not derived from raw response"):
validator.validate_run(run_dir, verify_latest=False)
def test_reproduction_pin_tampering_fails_even_after_resealing(tmp_path):
run_dir = copied_run(tmp_path)
path = run_dir / "reproduction_contract.json"
data = json.loads(path.read_text(encoding="utf-8"))
data["future_reproduction"]["vlm_source"]["original_revision"] = "0" * 40
write_json(path, data)
reseal_artifact(run_dir, path.name)
with pytest.raises(validator.EvidenceError, match="reproduction contract"):
validator.validate_run(run_dir, verify_latest=False)
chapter8/MiniMind-pretrain/validation/validate_evidence.py
+313
View File
@@ -0,0 +1,313 @@
#!/usr/bin/env python3
"""Fail-closed validator for Experiment 8-3 retained training evidence."""
from __future__ import annotations
import argparse
import hashlib
import json
import re
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
EXPERIMENT_DIR = HERE.parent
REPO_ROOT = EXPERIMENT_DIR.parents[1]
LATEST_PATH = HERE / "latest.json"
ARMS = {"original", "qk_norm_muon"}
STAGES = {"pretrain", "sft", "dpo"}
EXPECTED_COUNTS = {
("original", "pretrain"): 7,
("original", "sft"): 8,
("original", "dpo"): 9,
("qk_norm_muon", "pretrain"): 7,
("qk_norm_muon", "sft"): 9,
("qk_norm_muon", "dpo"): 9,
}
EXPECTED_SOURCE_REVISION = "8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795"
EXPECTED_DATASET_REVISION = "84983ed4dec7836d240577760c1d6be5d4cabcf9"
EXPECTED_DATASET_FILES = {
"pretrain_hq.jsonl": (
"9801b0d2210c61c2e4bc130f6dc4b3c870698a88d04af8f103c23dd5f0ce2440",
1_669_750_047,
),
"sft_512.jsonl": (
"053b7d09574e48a86232e929211434ff9e5016c6ed13312e63687dd52edcbebf",
7_531_517_862,
),
"dpo.jsonl": (
"ee934a8a455ccc99d1334d63e1254dd1d64f497fd067cfcbb71e3043f5b46768",
53_653_322,
),
}
SECRET_PATTERNS = (
re.compile(r"(?i)authorization\s*[:=]\s*bearer\s+\S+"),
re.compile(r"(?i)(?:api[_-]?key|secret)\s*[:=]\s*[A-Za-z0-9._-]{16,}"),
re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b"),
)
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_json(path: Path) -> dict[str, Any]:
value = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(value, dict):
raise AssertionError(f"{path} must contain a JSON object")
return value
def resolve_relative(base: Path, relative: str) -> Path:
path = (base / relative).resolve()
if not path.is_relative_to(base.resolve()):
raise AssertionError(f"path escapes evidence root: {relative}")
return path
def check_record(path: Path, record: dict[str, Any]) -> None:
if not path.is_file():
raise AssertionError(f"missing declared file: {path}")
if path.stat().st_size != record.get("bytes"):
raise AssertionError(f"byte count mismatch: {path}")
if sha256_file(path) != record.get("sha256"):
raise AssertionError(f"SHA-256 mismatch: {path}")
def parse_response_content(content: str) -> dict[str, Any]:
stripped = content.strip()
if stripped.startswith("```"):
stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
stripped = re.sub(r"\s*```$", "", stripped)
value = json.loads(stripped)
if not isinstance(value, dict):
raise AssertionError("judge response content must decode to an object")
return value
def find_pair(
retained: dict[str, Any], arm: str, stage: str, keyword: str
) -> dict[str, str]:
cells = [
cell
for cell in retained["cells"]
if cell.get("arm") == arm and cell.get("stage") == stage
]
if len(cells) != 1:
raise AssertionError(f"missing or duplicate retained cell: {arm}/{stage}")
matches = [
pair
for pair in cells[0]["pairs"]
if keyword.lower() in pair.get("prompt", "").lower()
]
if len(matches) != 1:
raise AssertionError(f"missing or duplicate selected prompt: {arm}/{stage}/{keyword}")
return matches[0]
def validate(latest_path: Path = LATEST_PATH) -> dict[str, Any]:
latest = load_json(latest_path)
if latest.get("experiment") != "8-3" or latest.get("status") != "passed":
raise AssertionError("latest pointer is not a passed Experiment 8-3 run")
run_dir = resolve_relative(EXPERIMENT_DIR, latest["run_dir"])
manifest_path = run_dir / "manifest.json"
if sha256_file(manifest_path) != latest.get("manifest_sha256"):
raise AssertionError("latest manifest hash mismatch")
manifest = load_json(manifest_path)
if manifest.get("run_id") != latest.get("run_id"):
raise AssertionError("run ID mismatch between latest and manifest")
if manifest.get("experiment") != "8-3" or manifest.get("status") != "passed":
raise AssertionError("manifest is not a passed Experiment 8-3 run")
if manifest.get("checkpoint_policy") != "not distributed; not an acceptance artifact":
raise AssertionError("manifest checkpoint policy is incorrect")
inputs = manifest.get("inputs")
artifacts = manifest.get("artifacts")
if not isinstance(inputs, list) or len(inputs) != 5:
raise AssertionError("manifest must bind exactly five repository inputs")
if not isinstance(artifacts, list) or len(artifacts) != 5:
raise AssertionError("manifest must bind exactly five run artifacts")
for record in inputs:
check_record(resolve_relative(REPO_ROOT, record["path"]), record)
for record in artifacts:
check_record(resolve_relative(run_dir, record["path"]), record)
retained = load_json(run_dir / "retained_outputs.json")
report_record = next(
(record for record in inputs if record.get("path") == retained.get("source_report")),
None,
)
if report_record is None or retained.get("source_report_sha256") != report_record.get("sha256"):
raise AssertionError("retained source-report hash does not match the manifest input")
if retained.get("cell_count") != 6 or retained.get("output_count") != 49:
raise AssertionError("retained report must contain six cells and 49 outputs")
if set(retained.get("arms", [])) != ARMS or set(retained.get("stages", [])) != STAGES:
raise AssertionError("retained report arm/stage coverage is incomplete")
cells = retained.get("cells")
if not isinstance(cells, list) or len(cells) != 6:
raise AssertionError("retained cells are malformed")
combos: set[tuple[str, str]] = set()
for cell in cells:
combo = (cell.get("arm"), cell.get("stage"))
if combo in combos or combo not in EXPECTED_COUNTS:
raise AssertionError(f"duplicate or unexpected cell: {combo}")
combos.add(combo)
pairs = cell.get("pairs")
if not isinstance(pairs, list) or len(pairs) != EXPECTED_COUNTS[combo]:
raise AssertionError(f"wrong retained pair count for {combo}")
if any(not pair.get("prompt") or not pair.get("output") for pair in pairs):
raise AssertionError(f"empty retained prompt/output in {combo}")
if combos != set(EXPECTED_COUNTS):
raise AssertionError("not all arm/stage cells are present")
receipts_root = load_json(run_dir / "judge_receipts.json")
if receipts_root.get("credential_headers_retained") is not False:
raise AssertionError("credential header retention must be explicitly false")
calls = receipts_root.get("calls")
if not isinstance(calls, list) or len(calls) != 8:
raise AssertionError("exactly eight raw judge calls are required")
response_ids: set[str] = set()
normalized_rows: dict[str, Any] = {}
for expected_case_id, call in enumerate(calls, start=1):
if call.get("case_id") != expected_case_id or call.get("http_status") != 200:
raise AssertionError("judge calls must be successful and ordered by case ID")
if call.get("credential_headers_retained") is not False:
raise AssertionError("per-call credential retention boundary is missing")
response_id = call.get("response_id")
if not isinstance(response_id, str) or not response_id or response_id in response_ids:
raise AssertionError("judge response IDs must be present and unique")
response_ids.add(response_id)
if call.get("latency_ms", 0) <= 0 or call.get("usage", {}).get("total_tokens", 0) <= 0:
raise AssertionError("judge usage and positive latency must be retained")
raw_response = call.get("response", {})
if raw_response.get("id") != response_id or raw_response.get("usage") != call.get("usage"):
raise AssertionError("copied response ID/usage does not match raw response")
try:
content = raw_response["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError) as exc:
raise AssertionError("raw judge response is missing message content") from exc
judgment = parse_response_content(content)
if judgment != call.get("judgment"):
raise AssertionError("normalized judgment does not match raw response content")
if str(judgment.get("case_id")) != str(expected_case_id):
raise AssertionError("raw judgment has the wrong case ID")
if set(judgment.get("candidates", {})) != {"A", "B"}:
raise AssertionError("judge judgment must score A and B")
if judgment.get("winner") not in {"A", "B", "tie"}:
raise AssertionError("judge winner is invalid")
mapping = call.get("blind_map")
if not isinstance(mapping, dict) or set(mapping) != {"A", "B"} or set(mapping.values()) != ARMS:
raise AssertionError("blind mapping must cover both arms")
request = call.get("request")
request_text = json.dumps(request, ensure_ascii=False).lower()
if "qk_norm_muon" in request_text or '"original"' in request_text:
raise AssertionError("judge request leaks an arm identity")
try:
user_payload = json.loads(request["messages"][1]["content"])
request_candidates = user_payload["candidates"]
except (KeyError, IndexError, TypeError, json.JSONDecodeError) as exc:
raise AssertionError("judge request is missing structured candidates") from exc
if user_payload.get("case_id") != expected_case_id:
raise AssertionError("judge request case ID mismatch")
for label, arm in mapping.items():
retained_pair = find_pair(retained, arm, call["stage"], call["keyword"])
expected_candidate = {
"historical_prompt": retained_pair["prompt"],
"historical_output": retained_pair["output"],
}
if request_candidates.get(label) != expected_candidate:
raise AssertionError("raw judge request is not bound to the retained output")
normalized_rows[str(expected_case_id)] = {
mapping[label]: score for label, score in judgment["candidates"].items()
}
contract = load_json(run_dir / "reproduction_contract.json")
future = contract.get("future_reproduction", {})
source = future.get("source", {})
dataset = future.get("dataset", {})
if source.get("revision") != EXPECTED_SOURCE_REVISION:
raise AssertionError("frozen MiniMind source revision mismatch")
source_hashes = source.get("file_sha256")
if not isinstance(source_hashes, dict) or len(source_hashes) < 12:
raise AssertionError("frozen source file hashes are incomplete")
if any(not re.fullmatch(r"[0-9a-f]{64}", value) for value in source_hashes.values()):
raise AssertionError("invalid frozen source SHA-256")
if dataset.get("revision") != EXPECTED_DATASET_REVISION:
raise AssertionError("frozen dataset revision mismatch")
dataset_files = dataset.get("files", {})
for name, (expected_hash, expected_bytes) in EXPECTED_DATASET_FILES.items():
record = dataset_files.get(name, {})
if record.get("lfs_sha256") != expected_hash or record.get("bytes") != expected_bytes:
raise AssertionError(f"frozen dataset file mismatch: {name}")
if len(future.get("commands", {})) != 6:
raise AssertionError("all six reproduction commands are required")
boundary = contract.get("historical_evidence_boundary", {})
for key in (
"historical_source_revision_retained",
"historical_dataset_hashes_retained",
"historical_checkpoint_hashes_retained",
"historical_stepwise_training_logs_retained",
):
if boundary.get(key) is not False:
raise AssertionError(f"historical provenance boundary is not explicit: {key}")
mechanisms = contract.get("model_and_training", {}).get("source_verified_mechanisms", {})
if not mechanisms or not all(value is True for value in mechanisms.values()):
raise AssertionError("source mechanism assertions are incomplete")
policy = contract.get("checkpoint_policy", {})
if policy.get("distributed_with_book") is not False or policy.get("acceptance_artifact") is not False:
raise AssertionError("checkpoint policy does not match the book contract")
summary = load_json(run_dir / "summary.json")
if summary.get("status") != "passed" or summary.get("acceptance", {}).get("passed") is not True:
raise AssertionError("summary acceptance did not pass")
acceptance = summary["acceptance"]
if not all(value is True for key, value in acceptance.items() if key != "passed"):
failed = [key for key, value in acceptance.items() if key != "passed" and value is not True]
raise AssertionError(f"required acceptance gates failed: {failed}")
if summary.get("per_case_arm_scores") != normalized_rows:
raise AssertionError("summary scores do not match raw judge responses")
findings = summary.get("scientific_findings", {})
if not isinstance(findings.get("blind_judge_prefers_qk_norm_muon_overall"), bool):
raise AssertionError("comparative scientific finding is missing")
if findings.get("reported_loss_comparison_retained_but_not_independently_recomputed") is not True:
raise AssertionError("loss-evidence qualification is missing")
for record in artifacts:
path = resolve_relative(run_dir, record["path"])
if path.suffix not in {".json", ".md"}:
continue
text = path.read_text(encoding="utf-8")
for pattern in SECRET_PATTERNS:
if pattern.search(text):
raise AssertionError(f"possible credential in retained artifact: {path.name}")
return {
"experiment": "8-3",
"run_id": latest["run_id"],
"status": "passed",
"inputs_verified": len(inputs),
"artifacts_verified": len(artifacts),
"judge_receipts_verified": len(calls),
"outputs_verified": retained["output_count"],
"manifest_sha256": latest["manifest_sha256"],
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--latest", type=Path, default=LATEST_PATH)
args = parser.parse_args()
result = validate(args.latest.resolve())
print(json.dumps(result, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())
chapter8/MiniMind-pretrain/validation/validate_vlm_evidence.py
Executable
+359
View File
@@ -0,0 +1,359 @@
#!/usr/bin/env python3
"""Fail-closed validator for the canonical Experiment 8-4 evidence package."""
from __future__ import annotations
import argparse
import base64
import hashlib
import json
import re
from pathlib import Path
from typing import Any
import run_vlm_training_report_audit as audit
HERE = Path(__file__).resolve().parent
EXPERIMENT_DIR = HERE.parent
REPO_ROOT = EXPERIMENT_DIR.parents[1]
EXPECTED_ARTIFACTS = {
"retained_outputs.json",
"reproduction_contract.json",
"judge_receipts.json",
"summary.json",
"report.md",
}
FORBIDDEN_SECRET_PATTERNS = (
re.compile(r"(?i)authorization\s*[:=]\s*bearer"),
re.compile(r"(?i)api[_-]?key\s*[:=]\s*['\"]?[A-Za-z0-9_-]{16,}"),
re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b"),
)
class EvidenceError(RuntimeError):
pass
def fail(message: str) -> None:
raise EvidenceError(message)
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_json(path: Path) -> Any:
try:
return json.loads(path.read_text(encoding="utf-8"))
except (OSError, json.JSONDecodeError) as exc:
fail(f"invalid JSON {path}: {exc}")
def resolve_repo_path(relative: str) -> Path:
path = (REPO_ROOT / relative).resolve()
if not path.is_relative_to(REPO_ROOT.resolve()):
fail(f"input escapes repository: {relative}")
return path
def verify_records(
records: list[dict[str, Any]], *, base: Path, expected_names: set[str] | None = None
) -> None:
names = []
for record in records:
if set(record) != {"path", "sha256", "bytes"}:
fail(f"malformed hash record: {record}")
relative = record["path"]
if not isinstance(relative, str) or not relative:
fail("hash record path is missing")
path = (base / relative).resolve()
if not path.is_relative_to(base.resolve()):
fail(f"hash record escapes base: {relative}")
if not path.is_file() or path.is_symlink():
fail(f"hashed file missing or symlinked: {path}")
if path.stat().st_size != record["bytes"]:
fail(f"byte count mismatch: {path}")
if sha256_file(path) != record["sha256"]:
fail(f"SHA-256 mismatch: {path}")
names.append(relative)
if len(names) != len(set(names)):
fail("duplicate hash records")
if expected_names is not None and set(names) != expected_names:
fail(f"artifact set mismatch: {set(names)} != {expected_names}")
def decode_image_url(url: str) -> bytes:
match = re.fullmatch(r"data:image/[A-Za-z0-9.+-]+;base64,([A-Za-z0-9+/=]+)", url)
if not match:
fail("judge request does not contain an exact base64 image data URL")
try:
return base64.b64decode(match.group(1), validate=True)
except ValueError as exc:
fail(f"invalid image base64: {exc}")
def validate_receipts(
retained: dict[str, Any], receipts_doc: dict[str, Any], summary: dict[str, Any]
) -> list[dict[str, Any]]:
if (
receipts_doc.get("schema_version") != "exp8-4-judge-receipts-v1"
or receipts_doc.get("experiment") != "8-4"
):
fail("wrong judge receipt schema or experiment")
if receipts_doc.get("credential_headers_retained") is not False:
fail("judge receipt must state that credential headers were not retained")
receipts = receipts_doc.get("calls")
if not isinstance(receipts, list) or [row.get("image") for row in receipts] != list(
audit.IMAGE_FILES
):
fail("judge calls must cover the eight images exactly in canonical order")
response_ids: list[str] = []
for receipt in receipts:
image = receipt["image"]
if receipt.get("image_source_filename") != audit.IMAGE_FILES[image]:
fail(f"wrong source image filename for {image}")
if receipt.get("image_sha256") != audit.IMAGE_SHA256[image]:
fail(f"wrong image SHA-256 for {image}")
if not isinstance(receipt.get("image_bytes"), int) or receipt["image_bytes"] <= 0:
fail(f"missing image byte count for {image}")
if receipt.get("provider") != "ark" or receipt.get("credential_env") != "ARK_API_KEY":
fail(f"wrong provider metadata for {image}")
if receipt.get("credential_headers_retained") is not False:
fail(f"credential header retention is not false for {image}")
if (
receipt.get("http_status") != 200
or not isinstance(receipt.get("latency_ms"), (int, float))
or receipt["latency_ms"] <= 0
):
fail(f"invalid transport evidence for {image}")
if receipt.get("blind_seed") != audit.BLIND_SEED or receipt.get(
"blind_map"
) != audit.blind_mapping(image):
fail(f"blind mapping mismatch for {image}")
if set(receipt["blind_map"]) != set(audit.LABELS) or set(
receipt["blind_map"].values()
) != set(audit.CONFIGS):
fail(f"blind map is not a bijection for {image}")
request = receipt.get("request")
if not isinstance(request, dict) or request.get("temperature") != 0:
fail(f"malformed deterministic judge request for {image}")
if request.get("response_format") != {"type": "json_object"}:
fail(f"judge request is not fail-closed JSON mode for {image}")
if request.get("model") != summary["judge"]["model"]:
fail(f"judge model mismatch for {image}")
messages = request.get("messages")
if (
not isinstance(messages, list)
or len(messages) != 2
or messages[0].get("role") != "system"
):
fail(f"malformed judge messages for {image}")
user_content = messages[1].get("content")
if not isinstance(user_content, list) or len(user_content) != 2:
fail(f"image-aware user content missing for {image}")
image_part, text_part = user_content
if (
image_part.get("type") != "image_url"
or image_part.get("image_url", {}).get("detail") != "high"
):
fail(f"high-detail image input missing for {image}")
raw_image = decode_image_url(image_part["image_url"].get("url", ""))
if (
hashlib.sha256(raw_image).hexdigest() != audit.IMAGE_SHA256[image]
or len(raw_image) != receipt["image_bytes"]
):
fail(f"request image bytes do not match pinned input for {image}")
if text_part.get("type") != "text" or not isinstance(text_part.get("text"), str):
fail(f"judge text input missing for {image}")
try:
prompt = json.loads(text_part["text"])
except json.JSONDecodeError as exc:
fail(f"judge text is not canonical JSON for {image}: {exc}")
expected_outputs = audit.outputs_for_image(retained, image)
expected_candidates = {
label: expected_outputs[config] for label, config in receipt["blind_map"].items()
}
if prompt.get("image") != image or prompt.get("candidates") != expected_candidates:
fail(f"anonymous candidate text does not match retained outputs for {image}")
prompt_text = json.dumps(prompt, ensure_ascii=False)
if any(config in prompt_text for config in audit.CONFIGS):
fail(f"judge prompt leaks configuration identity for {image}")
response = receipt.get("response")
response_id = receipt.get("response_id")
if (
not isinstance(response, dict)
or response.get("id") != response_id
or not isinstance(response_id, str)
or not response_id
):
fail(f"raw response ID mismatch for {image}")
if (
response.get("usage") != receipt.get("usage")
or not isinstance(receipt.get("usage", {}).get("total_tokens"), int)
or receipt["usage"]["total_tokens"] <= 0
):
fail(f"raw response usage mismatch for {image}")
try:
parsed = audit.extract_json_object(response["choices"][0]["message"]["content"])
audit.validate_judgment(parsed, image)
except (KeyError, TypeError, ValueError, json.JSONDecodeError) as exc:
fail(f"invalid raw judgment for {image}: {exc}")
if parsed != receipt.get("judgment"):
fail(f"normalized judgment is not derived from raw response for {image}")
response_ids.append(response_id)
if len(set(response_ids)) != len(audit.IMAGE_FILES):
fail("judge response IDs are not unique")
return receipts
def scan_credentials(run_dir: Path) -> None:
for path in run_dir.iterdir():
if not path.is_file() or path.suffix not in {".json", ".md"}:
continue
text = path.read_text(encoding="utf-8")
for pattern in FORBIDDEN_SECRET_PATTERNS:
if pattern.search(text):
fail(f"possible credential material in {path.name}")
def validate_run(run_dir: Path, *, verify_latest: bool = True) -> dict[str, Any]:
run_dir = run_dir.resolve()
if not run_dir.is_dir():
fail(f"missing run directory: {run_dir}")
if any(path.is_symlink() for path in run_dir.iterdir()):
fail("run directory contains a symlink")
manifest = load_json(run_dir / "manifest.json")
if (
manifest.get("schema_version") != "exp8-4-manifest-v1"
or manifest.get("experiment") != "8-4"
):
fail("wrong manifest schema or experiment")
if (
manifest.get("status") != "passed"
or manifest.get("checkpoint_policy") != "not distributed; not an acceptance artifact"
):
fail("manifest does not declare a passed checkpoint-free report")
if not isinstance(manifest.get("inputs"), list) or not isinstance(
manifest.get("artifacts"), list
):
fail("manifest hash lists are missing")
for record in manifest["inputs"]:
path = resolve_repo_path(record.get("path", ""))
if (
not path.is_file()
or path.is_symlink()
or path.stat().st_size != record.get("bytes")
or sha256_file(path) != record.get("sha256")
):
fail(f"input hash mismatch: {record.get('path')}")
if len({record["path"] for record in manifest["inputs"]}) != len(manifest["inputs"]):
fail("duplicate manifest inputs")
verify_records(manifest["artifacts"], base=run_dir, expected_names=EXPECTED_ARTIFACTS)
retained = load_json(run_dir / "retained_outputs.json")
current_retained = audit.parse_retained_outputs()
if retained != current_retained:
fail("retained outputs do not exactly match the content-hashed book report")
if retained.get("cell_count") != 8 or retained.get("output_count") != 64:
fail("retained output coverage is incomplete")
if retained.get("configs") != list(audit.CONFIGS) or retained.get("images") != list(
audit.IMAGE_FILES
):
fail("retained configuration/image contract changed")
contract = load_json(run_dir / "reproduction_contract.json")
if contract != audit.reproduction_contract():
fail("reproduction contract differs from frozen source/data/model pins")
if contract["checkpoint_policy"]["acceptance_artifact"] is not False:
fail("checkpoint policy was weakened")
summary = load_json(run_dir / "summary.json")
if summary.get("schema_version") != "exp8-4-summary-v1" or summary.get("status") != "passed":
fail("summary is not a passed Experiment 8-4 report")
receipts_doc = load_json(run_dir / "judge_receipts.json")
receipts = validate_receipts(retained, receipts_doc, summary)
recomputed = audit.summarize(retained, receipts, contract)
if summary != recomputed:
fail("summary metrics or acceptance gates do not recompute exactly")
if manifest.get("acceptance") != summary.get("acceptance") or not all(
summary["acceptance"].values()
):
fail("manifest/summary acceptance mismatch")
report = (run_dir / "report.md").read_text(encoding="utf-8")
if (
"Status: **passed**" not in report
or summary["scientific_findings"]["top_configuration"] not in report
):
fail("rendered report does not bind the recomputed result")
scan_credentials(run_dir)
if verify_latest:
latest = load_json(audit.LATEST_PATH)
if latest.get("schema_version") != "exp8-4-latest-v1" or latest.get("experiment") != "8-4":
fail("latest pointer has wrong schema or experiment")
if latest.get("run_id") != manifest.get("run_id") or latest.get("status") != "passed":
fail("latest pointer does not identify this passed run")
if latest.get("manifest_sha256") != sha256_file(run_dir / "manifest.json"):
fail("latest pointer manifest SHA-256 mismatch")
expected_run_dir = EXPERIMENT_DIR / latest.get("run_dir", "")
if expected_run_dir.resolve() != run_dir:
fail("latest pointer resolves to another run")
return {
"experiment": "8-4",
"status": "passed",
"run_id": manifest["run_id"],
"cells": retained["cell_count"],
"outputs": retained["output_count"],
"images": len(audit.IMAGE_FILES),
"judge_receipts": len(receipts),
"artifacts": len(manifest["artifacts"]),
"inputs": len(manifest["inputs"]),
"manifest_sha256": sha256_file(run_dir / "manifest.json"),
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--run-dir", type=Path, help="Run directory; defaults to validation/latest_vlm.json"
)
parser.add_argument(
"--no-latest",
action="store_true",
help="Skip latest-pointer binding (for deliberate tamper tests only)",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
if args.run_dir is None:
latest = load_json(audit.LATEST_PATH)
run_dir = EXPERIMENT_DIR / latest.get("run_dir", "")
else:
run_dir = args.run_dir
try:
result = validate_run(run_dir, verify_latest=not args.no_latest)
except EvidenceError as exc:
print(
json.dumps(
{"experiment": "8-4", "status": "failed", "error": str(exc)},
indent=2,
sort_keys=True,
)
)
return 1
print(json.dumps(result, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())
chapter8/MultilingualReasoning/README.md
+1133
View File
File diff suppressed because it is too large Load Diff
chapter8/MultilingualReasoning/evaluate_multilingual.py
+587
View File
@@ -0,0 +1,587 @@
"""Multilingual Reasoning Evaluator for AI Agent Book (Chapter 7).
Evaluates LLM reasoning models across multiple languages (English, Spanish,
French, Chinese, Japanese) measuring CoT language fidelity, task accuracy,
token usage, and cross-lingual transfer efficiency.
"""
from __future__ import annotations
import inspect
import math
import re
import statistics
import warnings
from typing import Any, Callable, Dict, List, Optional, Sequence, Union
LANG_MAP: dict[str, str] = {
"en": "English",
"english": "English",
"es": "Spanish",
"spanish": "Spanish",
"fr": "French",
"french": "French",
"zh": "Chinese",
"chinese": "Chinese",
"zh-cn": "Chinese",
"zh-tw": "Chinese",
"ja": "Japanese",
"japanese": "Japanese",
"de": "German",
"german": "German",
"it": "Italian",
"italian": "Italian",
"pt": "Portuguese",
"portuguese": "Portuguese",
"ru": "Russian",
"russian": "Russian",
"ko": "Korean",
"korean": "Korean",
"ar": "Arabic",
"arabic": "Arabic",
"hi": "Hindi",
"hindi": "Hindi",
"nl": "Dutch",
"dutch": "Dutch",
"tr": "Turkish",
"turkish": "Turkish",
}
# Regex patterns for script detection
CJK_RE = re.compile(r"[\u4e00-\u9fff\u3400-\u4dbf]")
HIRAGANA_RE = re.compile(r"[\u3040-\u309f]")
KATAKANA_RE = re.compile(r"[\u30a0-\u30ff]")
SPANISH_SPECIAL_RE = re.compile(r"[áéíóúüñÁÉÍÓÚÜÑ¿¡]")
FRENCH_SPECIAL_RE = re.compile(r"[éèêëàâùûçôîïÉÈÊËÀÂÙÛÇÔÎÏ]")
SPANISH_WORDS = {
"el", "la", "los", "las", "un", "una", "de", "en", "que", "es", "por", "para",
"con", "del", "al", "como", "más", "mas", "pero", "sus", "porque", "entonces",
"paso", "respuesta", "solucion", "solución", "por lo tanto", "primero", "luego"
}
FRENCH_WORDS = {
"le", "la", "les", "un", "une", "des", "du", "de", "en", "et", "est", "que",
"qui", "pour", "dans", "ce", "sur", "avec", "plus", "par", "mais", "donc",
"parce que", "alors", "etape", "étape", "reponse", "réponse", "solution", "premièrement"
}
ENGLISH_WORDS = {
"the", "be", "to", "of", "and", "a", "in", "that", "have", "it", "for",
"not", "on", "with", "he", "as", "you", "do", "at", "this", "but", "his",
"by", "from", "they", "we", "say", "her", "she", "or", "an", "will", "my",
"one", "all", "would", "there", "their", "what", "so", "up", "out", "if",
"about", "who", "get", "which", "go", "me", "when", "make", "can", "like",
"time", "no", "just", "him", "know", "take", "people", "into", "year",
"your", "good", "some", "could", "them", "see", "other", "than", "then",
"now", "look", "only", "come", "its", "over", "think", "also", "back",
"after", "use", "two", "how", "our", "work", "first", "well", "way",
"even", "new", "want", "because", "any", "these", "give", "day", "most",
"us", "therefore", "step", "reasoning", "solution", "answer", "equals",
"is", "plus", "minus", "times", "divided", "equal", "result"
}
def normalize_language(lang: str) -> str:
"""Normalize language identifier string to canonical English name."""
cleaned = str(lang).strip().lower()
canonical = LANG_MAP.get(cleaned)
if canonical is not None:
return canonical
# Unknown language: title-case for consistent multi-word names and warn
# so callers notice silent filtering in evaluate().
title_cased = cleaned.title()
warnings.warn(
f"Unrecognized language identifier {lang!r}; normalized to {title_cased!r}. "
f"Add it to LANG_MAP for reliable matching.",
stacklevel=2,
)
return title_cased
def estimate_tokens(text: str) -> int:
"""Estimate token count for a string if exact count is unavailable."""
if not text:
return 0
cjk_count = len(CJK_RE.findall(text)) + len(HIRAGANA_RE.findall(text)) + len(KATAKANA_RE.findall(text))
non_cjk_text = CJK_RE.sub(" ", HIRAGANA_RE.sub(" ", KATAKANA_RE.sub(" ", text)))
words = non_cjk_text.split()
# ~1.3 tokens per word for Latin scripts, ~1.5 tokens per character for CJK/Kana
return max(1, int(len(words) * 1.3 + cjk_count * 1.5))
def _extract_token_counts(tu: Any) -> Optional[dict[str, Optional[int]]]:
if tu is None:
return None
def get_val(key1: str, key2: Optional[str] = None) -> Optional[int]:
val = None
if isinstance(tu, dict):
val = tu.get(key1)
if val is None and key2:
val = tu.get(key2)
else:
val = getattr(tu, key1, None)
if val is None and key2:
val = getattr(tu, key2, None)
if val is not None:
try:
return int(val)
except (ValueError, TypeError):
return None
return None
p_tok = get_val("prompt_tokens", "input_tokens")
c_tok = get_val("completion_tokens", "output_tokens")
r_tok = get_val("reasoning_tokens")
t_tok = get_val("total_tokens")
if any(x is not None for x in (p_tok, c_tok, r_tok, t_tok)):
return {
"prompt_tokens": p_tok,
"completion_tokens": c_tok,
"reasoning_tokens": r_tok,
"total_tokens": t_tok,
}
return None
def _accepts_language(fn: Any) -> bool:
try:
sig = inspect.signature(fn)
for param in sig.parameters.values():
if param.name == "language" or param.kind == inspect.Parameter.VAR_KEYWORD:
return True
return False
except (ValueError, TypeError):
return False
class MultilingualReasoningEvaluator:
"""Evaluates reasoning LLMs across multiple languages.
Computes:
- CoT Language Fidelity Score: Consistency of reasoning steps with target language.
- Task Accuracy: Correctness of final generated answers.
- Token Usage: Breakdown of prompt, completion, and reasoning token costs.
- Cross-Lingual Transfer Efficiency: Relative performance across non-English languages.
"""
def __init__(self, target_languages: Optional[Sequence[str]] = None) -> None:
if target_languages is None:
self.target_languages = ["English", "Spanish", "French", "Chinese", "Japanese"]
else:
self.target_languages = [normalize_language(lang) for lang in target_languages]
def evaluate_cot_fidelity(self, cot_text: str, target_language: str) -> float:
"""Calculate language fidelity score (0.0 - 1.0) for Chain-of-Thought text."""
if not cot_text or not cot_text.strip():
return 0.0
lang = normalize_language(target_language)
text = cot_text.strip()
non_space_chars = len(re.sub(r"\s+", "", text))
if non_space_chars == 0:
return 0.0
cjk_count = len(CJK_RE.findall(text))
hiragana_count = len(HIRAGANA_RE.findall(text))
katakana_count = len(KATAKANA_RE.findall(text))
kana_count = hiragana_count + katakana_count
if lang == "Chinese":
if kana_count > 0:
cjk_ratio = cjk_count / non_space_chars
return max(0.0, min(1.0, cjk_ratio / 0.3) * 0.7)
cjk_ratio = cjk_count / non_space_chars
return min(1.0, cjk_ratio / 0.35)
if lang == "Japanese":
if kana_count > 0:
j_ratio = (kana_count + cjk_count) / non_space_chars
return min(1.0, j_ratio / 0.35)
if cjk_count > 0:
return 0.4
return 0.0
# For European / Latin languages, CJK/Kana script implies cross-lingual leakage
if cjk_count > 0 or kana_count > 0:
return 0.0
words = [w.lower() for w in re.findall(r"\b[a-zA-ZáéíóúüñÁÉÍÓÚÜÑàâäèêëîïôöùûüçÀÂÄÉÈÊËÎÏÔÖÙÛÜÇ]+\b", text)]
total_words = len(words)
if total_words == 0:
return 1.0 if any(c.isalnum() for c in text) else 0.0
if lang == "Spanish":
special_count = len(SPANISH_SPECIAL_RE.findall(text))
spanish_word_matches = sum(1 for w in words if w in SPANISH_WORDS)
score = (special_count * 2 + spanish_word_matches) / max(1, total_words)
return min(1.0, max(0.2, score * 3.0))
if lang == "French":
special_count = len(FRENCH_SPECIAL_RE.findall(text))
french_word_matches = sum(1 for w in words if w in FRENCH_WORDS)
score = (special_count * 2 + french_word_matches) / max(1, total_words)
return min(1.0, max(0.2, score * 3.0))
if lang == "English":
english_word_matches = sum(1 for w in words if w in ENGLISH_WORDS)
french_special = len(FRENCH_SPECIAL_RE.findall(text))
spanish_special = len(SPANISH_SPECIAL_RE.findall(text))
penalty = (french_special + spanish_special) * 0.1
match_ratio = english_word_matches / max(1, total_words)
score = match_ratio * 1.8 - penalty
return min(1.0, max(0.0, score))
return 0.5
def evaluate_accuracy(self, predicted_answer: str, reference_answer: str) -> float:
"""Determine task accuracy (1.0 for match, 0.0 for mismatch)."""
pred = str(predicted_answer if predicted_answer is not None else "").strip().lower()
ref = str(reference_answer if reference_answer is not None else "").strip().lower()
if not pred or not ref:
return 0.0
if pred == ref:
return 1.0
# Strip common trailing punctuation
pred_clean = re.sub(r"[.,;!\?]+$", "", pred)
ref_clean = re.sub(r"[.,;!\?]+$", "", ref)
if pred_clean == ref_clean:
return 1.0
# Try numeric comparison
pred_nums = re.findall(r"[-+]?\d*\.?\d+", pred)
ref_nums = re.findall(r"[-+]?\d*\.?\d+", ref)
if pred_nums and ref_nums:
try:
p_val = float(pred_nums[-1])
r_val = float(ref_nums[-1])
if math.isclose(p_val, r_val, rel_tol=1e-4, abs_tol=1e-4):
return 1.0
except ValueError:
pass
# Substring matching for references with word boundaries
target_ref = ref_clean if ref_clean else ref
if target_ref:
# \b word boundaries don't work for CJK characters; use direct
# containment for non-ASCII references, and \b for Latin text.
if re.search(r"[^\x00-\x7f]", target_ref):
if target_ref in pred_clean or target_ref in pred:
return 1.0
else:
pattern = r"\b" + re.escape(target_ref) + r"\b"
if re.search(pattern, pred_clean) or re.search(pattern, pred):
return 1.0
return 0.0
def compute_token_usage(
self, prompt: str, reasoning: str, answer: str, model_output: Any = None
) -> dict[str, int]:
"""Extract or estimate prompt, completion, reasoning, and total token usage."""
tu = None
if isinstance(model_output, dict):
if "token_usage" in model_output and model_output["token_usage"] is not None:
tu = model_output["token_usage"]
else:
tu = model_output
elif model_output is not None:
if hasattr(model_output, "token_usage") and getattr(model_output, "token_usage") is not None:
tu = getattr(model_output, "token_usage")
else:
tu = model_output
extracted = _extract_token_counts(tu)
if extracted is not None:
r_tok = extracted["reasoning_tokens"] if extracted["reasoning_tokens"] is not None else estimate_tokens(reasoning)
p_tok = extracted["prompt_tokens"] if extracted["prompt_tokens"] is not None else estimate_tokens(prompt)
c_tok = extracted["completion_tokens"] if extracted["completion_tokens"] is not None else (r_tok + estimate_tokens(answer))
t_tok = extracted["total_tokens"] if extracted["total_tokens"] is not None else (p_tok + c_tok)
return {
"prompt_tokens": p_tok,
"completion_tokens": c_tok,
"reasoning_tokens": r_tok,
"total_tokens": t_tok,
}
p_tok = estimate_tokens(prompt)
r_tok = estimate_tokens(reasoning)
a_tok = estimate_tokens(answer)
c_tok = r_tok + a_tok
t_tok = p_tok + c_tok
return {
"prompt_tokens": p_tok,
"completion_tokens": c_tok,
"reasoning_tokens": r_tok,
"total_tokens": t_tok,
}
def _parse_model_output(self, raw_output: Any) -> tuple[str, str, Any]:
"""Parse raw model output into reasoning CoT, final answer, and token metadata."""
if isinstance(raw_output, dict):
reasoning = ""
for k in ("reasoning", "cot", "thinking"):
if k in raw_output and raw_output[k] is not None:
reasoning = str(raw_output[k])
break
answer = ""
for k in ("answer", "response", "predicted_answer"):
if k in raw_output and raw_output[k] is not None:
answer = str(raw_output[k])
break
tu = raw_output.get("token_usage")
return reasoning, answer, tu
if hasattr(raw_output, "reasoning") or hasattr(raw_output, "answer") or hasattr(raw_output, "cot") or hasattr(raw_output, "response") or hasattr(raw_output, "predicted_answer"):
reasoning = ""
for attr in ("reasoning", "cot", "thinking"):
if hasattr(raw_output, attr) and getattr(raw_output, attr) is not None:
reasoning = str(getattr(raw_output, attr))
break
answer = ""
for attr in ("answer", "response", "predicted_answer"):
if hasattr(raw_output, attr) and getattr(raw_output, attr) is not None:
answer = str(getattr(raw_output, attr))
break
tu = getattr(raw_output, "token_usage", None)
return reasoning, answer, tu
text = str(raw_output or "").strip()
# Handle <think>...</think> tags
if "<think>" in text and "</think>" in text:
parts = text.split("</think>", 1)
reasoning = parts[0].replace("<think>", "").strip()
answer = parts[1].strip()
return reasoning, answer, {}
# Handle Reasoning: / Answer: markers
if "reasoning:" in text.lower() and "answer:" in text.lower():
r_idx = text.lower().find("reasoning:")
a_idx = text.lower().find("answer:")
if r_idx < a_idx:
reasoning = text[r_idx + 10 : a_idx].strip()
answer = text[a_idx + 7 :].strip()
return reasoning, answer, {}
# If line breaks exist, treat first part as reasoning and last line as answer
lines = [line.strip() for line in text.split("\n") if line.strip()]
if len(lines) > 1:
reasoning = "\n".join(lines[:-1])
answer = lines[-1]
return reasoning, answer, {}
return text, text, {}
def _invoke_model(self, model: Any, prompt: str, language: str) -> Any:
"""Call model using appropriate signature (generate, predict, or call)."""
if callable(model):
if _accepts_language(model):
return model(prompt, language=language)
return model(prompt)
if hasattr(model, "generate") and callable(model.generate):
if _accepts_language(model.generate):
return model.generate(prompt, language=language)
return model.generate(prompt)
if hasattr(model, "predict") and callable(model.predict):
if _accepts_language(model.predict):
return model.predict(prompt, language=language)
return model.predict(prompt)
raise ValueError(f"Model object {type(model)} is not callable and lacks generate/predict methods.")
def evaluate_sample(self, model: Any, sample: dict[str, Any]) -> dict[str, Any]:
"""Evaluate a single dataset sample."""
lang_raw = None
for k in ("language", "target_language", "lang"):
if k in sample and sample[k] is not None:
lang_raw = sample[k]
break
language = normalize_language(str(lang_raw) if lang_raw is not None else "English")
prompt = ""
for k in ("prompt", "question", "input"):
if k in sample and sample[k] is not None:
prompt = str(sample[k])
break
reference_answer = ""
for k in ("reference_answer", "expected_answer", "ground_truth", "target", "answer"):
if k in sample and sample[k] is not None:
reference_answer = str(sample[k])
break
raw_output = self._invoke_model(model, prompt, language)
reasoning, answer, tu_raw = self._parse_model_output(raw_output)
cot_fidelity = self.evaluate_cot_fidelity(reasoning, language)
accuracy = self.evaluate_accuracy(answer, reference_answer)
token_usage = self.compute_token_usage(prompt, reasoning, answer, tu_raw or raw_output)
return {
"language": language,
"prompt": prompt,
"reference_answer": reference_answer,
"reasoning": reasoning,
"predicted_answer": answer,
"cot_fidelity": cot_fidelity,
"accuracy": accuracy,
"token_usage": token_usage,
}
def compute_transfer_efficiency(self, by_language_metrics: dict[str, dict[str, Any]]) -> dict[str, float]:
"""Calculate cross-lingual transfer efficiency relative to English.
If English accuracy is missing or zero, the highest per-language accuracy
becomes the reference. If no positive reference exists, efficiency is 0.0.
"""
raw_eng = by_language_metrics.get("English") if isinstance(by_language_metrics.get("English"), dict) else {}
english_acc = (raw_eng.get("accuracy") or 0.0) if raw_eng else 0.0
positive_accs = [
m["accuracy"]
for m in by_language_metrics.values()
if isinstance(m, dict) and m.get("accuracy") is not None and m.get("accuracy", 0.0) > 0
]
reference_acc = english_acc if english_acc > 0 else (max(positive_accs) if positive_accs else 0.0)
efficiencies: dict[str, float] = {}
for lang, metrics in by_language_metrics.items():
acc = (metrics.get("accuracy") or 0.0) if isinstance(metrics, dict) else 0.0
if reference_acc > 0:
efficiencies[lang] = round(acc / reference_acc, 4)
else:
efficiencies[lang] = 0.0
return efficiencies
def evaluate(self, model: Any, dataset: Sequence[dict[str, Any]]) -> dict[str, Any]:
"""Evaluate model on dataset and compile comprehensive report."""
def _empty_report() -> dict[str, Any]:
return {
"overall_accuracy": 0.0,
"overall_cot_fidelity": 0.0,
"overall_transfer_efficiency": 0.0,
"total_token_usage": {
"prompt_tokens": 0,
"completion_tokens": 0,
"reasoning_tokens": 0,
"total_tokens": 0,
},
"by_language": {},
"num_samples": 0,
}
if not dataset:
return _empty_report()
if self.target_languages:
target_langs = set(self.target_languages)
dataset = [
s for s in dataset
if normalize_language(
s.get("language") or s.get("target_language") or s.get("lang") or "English"
) in target_langs
]
if not dataset:
return _empty_report()
sample_results = []
for sample in dataset:
try:
sample_results.append(self.evaluate_sample(model, sample))
except Exception as e:
lang_raw = sample.get("language") or sample.get("target_language") or sample.get("lang") or "English"
sample_results.append({
"language": normalize_language(str(lang_raw)),
"prompt": str(sample.get("prompt") or sample.get("question") or sample.get("input") or ""),
"reference_answer": str(sample.get("reference_answer") or sample.get("expected_answer") or ""),
"reasoning": "",
"predicted_answer": "",
"cot_fidelity": 0.0,
"accuracy": 0.0,
"error": str(e),
"token_usage": {"prompt_tokens": 0, "completion_tokens": 0, "reasoning_tokens": 0, "total_tokens": 0},
})
if not sample_results:
return _empty_report()
by_lang_samples: dict[str, list[dict[str, Any]]] = {}
for res in sample_results:
lang = res["language"]
by_lang_samples.setdefault(lang, []).append(res)
by_language_metrics: dict[str, dict[str, Any]] = {}
tot_p_tokens = 0
tot_c_tokens = 0
tot_r_tokens = 0
tot_t_tokens = 0
for lang, samples in by_lang_samples.items():
cnt = len(samples)
acc = sum(s["accuracy"] for s in samples) / cnt
fid = sum(s["cot_fidelity"] for s in samples) / cnt
p_tok = sum(s["token_usage"]["prompt_tokens"] for s in samples)
c_tok = sum(s["token_usage"]["completion_tokens"] for s in samples)
r_tok = sum(s["token_usage"]["reasoning_tokens"] for s in samples)
t_tok = sum(s["token_usage"]["total_tokens"] for s in samples)
tot_p_tokens += p_tok
tot_c_tokens += c_tok
tot_r_tokens += r_tok
tot_t_tokens += t_tok
by_language_metrics[lang] = {
"sample_count": cnt,
"accuracy": round(acc, 4),
"cot_fidelity": round(fid, 4),
"token_usage": {
"prompt_tokens": p_tok,
"completion_tokens": c_tok,
"reasoning_tokens": r_tok,
"total_tokens": t_tok,
},
}
transfer_efficiencies = self.compute_transfer_efficiency(by_language_metrics)
for lang, eff in transfer_efficiencies.items():
by_language_metrics[lang]["transfer_efficiency"] = eff
non_english_effs = [
eff for lang, eff in transfer_efficiencies.items() if lang != "English"
]
overall_transfer_eff = (
round(sum(non_english_effs) / len(non_english_effs), 4)
if non_english_effs
else 1.0
)
overall_accuracy = round(sum(s["accuracy"] for s in sample_results) / len(sample_results), 4)
overall_fidelity = round(sum(s["cot_fidelity"] for s in sample_results) / len(sample_results), 4)
return {
"overall_accuracy": overall_accuracy,
"overall_cot_fidelity": overall_fidelity,
"overall_transfer_efficiency": overall_transfer_eff,
"total_token_usage": {
"prompt_tokens": tot_p_tokens,
"completion_tokens": tot_c_tokens,
"reasoning_tokens": tot_r_tokens,
"total_tokens": tot_t_tokens,
},
"by_language": by_language_metrics,
"num_samples": len(sample_results),
}
def run_evaluation(
model: Any, dataset: Sequence[dict[str, Any]], target_languages: Optional[Sequence[str]] = None
) -> dict[str, Any]:
"""Entrypoint function to run multilingual reasoning evaluation."""
evaluator = MultilingualReasoningEvaluator(target_languages=target_languages)
return evaluator.evaluate(model, dataset)
chapter8/MultilingualReasoning/gpt_oss_20b_sft.py
+554
View File
@@ -0,0 +1,554 @@
"""
多语言推理模型微调脚本
本脚本展示如何使用 Hugging Face 的 TRL 库对 OpenAI 的 gpt-oss-20b 模型进行微调，
使其能够在多种语言中进行有效推理。
基于 OpenAI Cookbook 教程：
https://cookbook.openai.com/articles/gpt-oss/fine-tune-transfomers
作者: Edward Beeching, Quentin Gallouédec, Lewis Tunstall
修改: 适配为完整的 Python 脚本
⚠️ 硬件要求（重要！）:
- GPU: H100（80GB 显存）或更高配置
- 训练时间: H100 上约 18 分钟
- 使用 Mxfp4Config 量化和 LoRA 进行内存高效训练
功能特性:
- 使用 Mxfp4Config（针对 OpenAI 模型优化的 4-bit 浮点格式）
- 使用 LoRA 进行内存高效的微调（包括 MoE 专家层）
- 支持多语言推理（英语、西班牙语、法语、德语、意大利语等）
- 可以混合语言（用一种语言提问，用另一种语言推理）
- 所有超参数与 OpenAI Cookbook 教程完全一致
"""
import os
import argparse
try:
import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, Mxfp4Config
from peft import LoraConfig, PeftModel, get_peft_model
from trl import SFTTrainer, SFTConfig
except ImportError:
torch = None
load_dataset = None
AutoModelForCausalLM = AutoTokenizer = Mxfp4Config = None
LoraConfig = PeftModel = get_peft_model = None
SFTTrainer = SFTConfig = None
# ============================================================================
# 第一部分：数据集准备
# ============================================================================
def load_and_prepare_dataset():
"""
加载并准备多语言推理数据集
使用 HuggingFaceH4/Multilingual-Thinking 数据集，该数据集包含：
- 多种语言的推理链（思维链）
- 支持英语、西班牙语、法语、德语、意大利语等
Returns:
Dataset: 格式化后的训练数据集
"""
print("=" * 80)
print("步骤 1: 加载数据集")
print("=" * 80)
# 从 Hugging Face Hub 加载数据集
dataset = load_dataset("HuggingFaceH4/Multilingual-Thinking")
print(f"数据集加载完成！")
print(f"训练样本数: {len(dataset['train'])}")
print(f"数据集列: {dataset['train'].column_names}")
print(f"\n示例数据:")
print(dataset['train'][0])
return dataset['train']
def format_chat_template(example, tokenizer):
"""
格式化对话模板
将数据集中的消息格式化为模型可以理解的对话格式
Args:
example: 数据集中的一个样本
tokenizer: 分词器
Returns:
dict: 格式化后的样本
"""
# 应用聊天模板（带 messages 类型校验）
messages = example.get("messages")
if not isinstance(messages, list):
messages = []
example["text"] = tokenizer.apply_chat_template(
messages,
tokenize=False,
)
return example
# ============================================================================
# 第二部分：模型准备
# ============================================================================
def load_base_model(model_name="openai/gpt-oss-20b"):
"""
加载基础模型和分词器
使用 Mxfp4Config 进行量化，这是专门为 OpenAI 模型优化的 4-bit 浮点格式。
Args:
model_name: 模型名称或路径
Returns:
tuple: (model, tokenizer)
"""
print("\n" + "=" * 80)
print("步骤 2: 加载基础模型")
print("=" * 80)
# 加载分词器
print(f"加载分词器: {model_name}")
tokenizer = AutoTokenizer.from_pretrained(model_name)
# 设置 pad token（如果不存在）
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
# 配置 Mxfp4 量化（针对 OpenAI 模型优化）
print("使用 Mxfp4Config 量化...")
quantization_config = Mxfp4Config(dequantize=True)
# 配置模型加载参数
model_kwargs = {
"attn_implementation": "eager", # 注意力实现方式
"torch_dtype": torch.bfloat16, # 使用 bfloat16 提高效率
"quantization_config": quantization_config, # Mxfp4 量化配置
"use_cache": False, # 训练时禁用 KV 缓存
"device_map": "auto", # 自动分配设备
}
# 加载模型
print(f"加载模型: {model_name}")
print("这可能需要几分钟时间...")
model = AutoModelForCausalLM.from_pretrained(model_name, **model_kwargs)
print(f"模型加载完成！")
print(f"模型参数量: {model.num_parameters() / 1e9:.2f}B")
return model, tokenizer
def prepare_model_for_lora(model, lora_rank=8, lora_alpha=16):
"""
配置 LoRA（低秩适应）进行高效微调
LoRA 只训练少量参数，大大减少内存使用和训练时间。
针对 openai/gpt-oss-20b 的 MoE（混合专家）架构，除了注意力层外，
还需要特别指定 MLP 专家层进行训练。
Args:
model: 基础模型
lora_rank: LoRA 秩（默认 8，与官方教程一致）
lora_alpha: LoRA 缩放参数（默认 16）
Returns:
PeftModel: 配置了 LoRA 的模型
"""
print("\n" + "=" * 80)
print("步骤 3: 配置 LoRA")
print("=" * 80)
# LoRA 配置（与 OpenAI Cookbook 一致）
peft_config = LoraConfig(
r=lora_rank, # LoRA 秩
lora_alpha=lora_alpha, # LoRA 缩放参数
target_modules="all-linear", # 目标所有线性层
target_parameters=[ # MoE 专家层的特定参数
"7.mlp.experts.gate_up_proj",
"7.mlp.experts.down_proj",
"15.mlp.experts.gate_up_proj",
"15.mlp.experts.down_proj",
"23.mlp.experts.gate_up_proj",
"23.mlp.experts.down_proj",
],
)
print("LoRA 配置:")
print(f" - Rank: {lora_rank}")
print(f" - Alpha: {lora_alpha}")
print(f" - 目标模块: {peft_config.target_modules}")
print(f" - MoE 专家层参数: {len(peft_config.target_parameters)} 个")
# 应用 LoRA
model = get_peft_model(model, peft_config)
# 打印可训练参数统计
trainable_params = sum(p.numel() for p in model.parameters() if p.requires_grad)
total_params = sum(p.numel() for p in model.parameters())
trainable_percent = 100 * trainable_params / total_params
print(f"\n可训练参数统计:")
print(f" - 可训练参数: {trainable_params:,} ({trainable_percent:.2f}%)")
print(f" - 总参数: {total_params:,}")
return model
# ============================================================================
# 第三部分：训练
# ============================================================================
def train_model(model, tokenizer, dataset, output_dir="./gpt-oss-20b-multilingual-reasoner",
batch_size=4, num_epochs=1, learning_rate=2e-4, max_seq_length=2048):
"""
使用 SFTTrainer 训练模型
Args:
model: 配置了 LoRA 的模型
tokenizer: 分词器
dataset: 训练数据集
output_dir: 输出目录
batch_size: 批次大小（根据 GPU 显存调整，默认 4）
num_epochs: 训练轮数（默认 1）
learning_rate: 学习率（默认 2e-4）
max_seq_length: 最大序列长度
Returns:
SFTTrainer: 训练好的 trainer 对象
"""
print("\n" + "=" * 80)
print("步骤 4: 开始训练")
print("=" * 80)
# 训练参数配置（与 OpenAI Cookbook 完全一致）
training_args = SFTConfig(
learning_rate=learning_rate,
gradient_checkpointing=True,
num_train_epochs=num_epochs,
logging_steps=1,
per_device_train_batch_size=batch_size,
gradient_accumulation_steps=4,
max_length=max_seq_length,
warmup_ratio=0.03,
lr_scheduler_type="cosine_with_min_lr",
lr_scheduler_kwargs={"min_lr_rate": 0.1},
output_dir=output_dir,
report_to="trackio", # 设为 "trackio" 以启用实验跟踪
push_to_hub=False, # 设为 True 以自动推送到 Hub
)
print("训练配置:")
print(f" - 批次大小: {batch_size}")
print(f" - 梯度累积步数: {training_args.gradient_accumulation_steps}")
print(f" - 有效批次大小: {batch_size * training_args.gradient_accumulation_steps}")
print(f" - 训练轮数: {num_epochs}")
print(f" - 学习率: {learning_rate}")
print(f" - 学习率调度: {training_args.lr_scheduler_type}")
print(f" - 最大序列长度: {max_seq_length}")
print(f" - 输出目录: {output_dir}")
# 初始化 SFTTrainer
trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset,
processing_class=tokenizer,
)
# 开始训练
print("\n开始训练...")
print("⚠️ 在 H100 GPU 上训练约需 18 分钟")
print("-" * 80)
trainer.train()
print("\n" + "=" * 80)
print("训练完成！")
print("=" * 80)
return trainer
# ============================================================================
# 第四部分：保存和推送模型
# ============================================================================
def save_and_push_model(trainer, output_dir, push_to_hub=False, hub_model_id=None):
"""
保存模型并可选择推送到 Hugging Face Hub
Args:
trainer: 训练好的 trainer 对象
output_dir: 输出目录
push_to_hub: 是否推送到 Hub
hub_model_id: Hub 上的模型 ID
"""
print("\n" + "=" * 80)
print("步骤 5: 保存模型")
print("=" * 80)
# 保存模型到本地
print(f"保存模型到: {output_dir}")
trainer.save_model(output_dir)
print("模型保存完成！")
# 可选：推送到 Hugging Face Hub
if push_to_hub:
if hub_model_id is None:
raise ValueError("需要提供 hub_model_id 才能推送到 Hub")
print(f"\n推送模型到 Hugging Face Hub: {hub_model_id}")
# Trainer.push_to_hub 从 args.hub_model_id 取仓库名；不设置的话会
# 忽略用户传入的 --hub_model_id，推到 output_dir 同名的默认仓库。
trainer.args.hub_model_id = hub_model_id
trainer.push_to_hub(
dataset_name="HuggingFaceH4/Multilingual-Thinking",
)
print("模型已成功推送到 Hub！")
# ============================================================================
# 第五部分：推理
# ============================================================================
def load_trained_model(base_model_name, peft_model_path):
"""
加载训练好的模型进行推理
Args:
base_model_name: 基础模型名称
peft_model_path: LoRA 权重路径
Returns:
tuple: (model, tokenizer)
"""
print("\n" + "=" * 80)
print("加载训练好的模型进行推理")
print("=" * 80)
# 加载分词器
print(f"加载分词器: {base_model_name}")
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
# 加载基础模型
print(f"加载基础模型: {base_model_name}")
model_kwargs = {
"attn_implementation": "eager",
"torch_dtype": "auto",
"use_cache": True, # 推理时启用 KV 缓存
"device_map": "auto",
}
base_model = AutoModelForCausalLM.from_pretrained(base_model_name, **model_kwargs)
# 加载并合并 LoRA 权重
print(f"加载 LoRA 权重: {peft_model_path}")
model = PeftModel.from_pretrained(base_model, peft_model_path)
print("合并 LoRA 权重与基础模型...")
model = model.merge_and_unload()
print("模型加载完成！")
return model, tokenizer
def generate_response(model, tokenizer, reasoning_language, user_prompt,
max_new_tokens=512, temperature=0.6, format_output=True):
"""
生成多语言推理响应
Args:
model: 训练好的模型
tokenizer: 分词器
reasoning_language: 推理使用的语言
user_prompt: 用户提问
max_new_tokens: 最大生成 token 数
temperature: 采样温度（越高越随机）
format_output: 是否格式化输出（使用明显的标记）
Returns:
str: 生成的完整响应
"""
# 构建消息
system_prompt = f"reasoning language: {reasoning_language}"
messages = [
{"role": "system", "content": system_prompt},
{"role": "user", "content": user_prompt},
]
# 应用聊天模板
input_ids = tokenizer.apply_chat_template(
messages,
add_generation_prompt=True,
return_tensors="pt",
).to(model.device)
# 生成配置
gen_kwargs = {
"max_new_tokens": max_new_tokens,
"do_sample": True,
"temperature": temperature,
"top_p": None,
"top_k": None,
}
# 生成响应
print(f"\n生成响应...")
print(f"推理语言: {reasoning_language}")
print(f"用户提问: {user_prompt}")
with torch.no_grad():
output_ids = model.generate(input_ids, **gen_kwargs)
# 解码输出 - 保留特殊标记以便解析
response_with_tokens = tokenizer.batch_decode(output_ids, skip_special_tokens=False)[0]
print("-" * 80)
print(response_with_tokens)
print("-" * 80)
def run_inference_examples(model, tokenizer):
"""
运行多个推理示例
Args:
model: 训练好的模型
tokenizer: 分词器
"""
print("\n" + "=" * 80)
print("推理示例")
print("=" * 80)
# 示例 1: 西班牙语提问，德语推理
print("\n[示例 1: 西班牙语提问 + 德语推理]")
generate_response(
model, tokenizer,
reasoning_language="German",
user_prompt="¿Cuál es el capital de Australia?", # 澳大利亚的首都是什么？
format_output=True,
)
# 示例 2: 英语提问，中文推理
print("\n\n[示例 2: 英语提问 + 中文推理]")
generate_response(
model, tokenizer,
reasoning_language="Chinese",
user_prompt="What is the national symbol of Canada?",
format_output=True,
)
# 示例 3: 中文提问，中文推理
print("\n\n[示例 3: 中文提问 + 中文推理]")
generate_response(
model, tokenizer,
reasoning_language="Chinese",
user_prompt="求解 x^2 - 2x + 1 = 0 的根",
format_output=True,
)
# ============================================================================
# 主函数
# ============================================================================
def main():
"""主函数：完整的训练流程"""
parser = argparse.ArgumentParser(description="多语言推理模型微调")
parser.add_argument(
"--mode",
type=str,
choices=["train", "inference", "full"],
default="full",
help="运行模式: train（仅训练）, inference（仅推理）, full（完整流程）"
)
parser.add_argument("--model_name", type=str, default="openai/gpt-oss-20b", help="基础模型名称")
parser.add_argument("--output_dir", type=str, default="./gpt-oss-20b-multilingual-reasoner", help="输出目录")
parser.add_argument("--batch_size", type=int, default=4, help="训练批次大小（默认 4，与官方教程一致）")
parser.add_argument("--num_epochs", type=int, default=1, help="训练轮数（默认 1，与官方教程一致）")
parser.add_argument("--learning_rate", type=float, default=2e-4, help="学习率（默认 2e-4，与官方教程一致）")
parser.add_argument("--max_seq_length", type=int, default=2048, help="最大序列长度")
parser.add_argument("--lora_rank", type=int, default=8, help="LoRA 秩（默认 8，与官方教程一致）")
parser.add_argument("--lora_alpha", type=int, default=16, help="LoRA alpha")
parser.add_argument("--push_to_hub", action="store_true", default=False, help="推送模型到 Hugging Face Hub")
parser.add_argument("--hub_model_id", type=str, default=None, help="Hub 模型 ID")
args = parser.parse_args()
print("=" * 80)
print("多语言推理模型微调")
print("=" * 80)
print(f"模式: {args.mode}")
print(f"基础模型: {args.model_name}")
print(f"输出目录: {args.output_dir}")
# 训练模式
if args.mode in ["train", "full"]:
# 1. 加载数据集
dataset = load_and_prepare_dataset()
# 2. 加载基础模型（使用 Mxfp4Config 量化）
model, tokenizer = load_base_model(args.model_name)
# 3. 配置 LoRA
model = prepare_model_for_lora(model, args.lora_rank, args.lora_alpha)
# 4. 训练模型
trainer = train_model(
model,
tokenizer,
dataset,
output_dir=args.output_dir,
batch_size=args.batch_size,
num_epochs=args.num_epochs,
learning_rate=args.learning_rate,
max_seq_length=args.max_seq_length,
)
# 5. 保存模型
save_and_push_model(
trainer,
args.output_dir,
push_to_hub=args.push_to_hub,
hub_model_id=args.hub_model_id,
)
if args.mode == "full":
# full 模式继续跑推理：先释放训练占用的显存，
# 再按推理路径从 output_dir 重新加载已保存的模型。
del trainer
del model
torch.cuda.empty_cache()
print("\n训练完成！已释放训练显存，继续运行推理示例。")
else:
print("\n训练完成！建议重启内核以释放 GPU 显存后再进行推理。")
# 推理模式（inference 单独运行；full 在训练后接着运行）
if args.mode in ["inference", "full"]:
if not os.path.exists(args.output_dir):
print(f"错误: 未找到模型目录 {args.output_dir}")
print("请先运行训练或指定正确的模型路径")
return
# 加载训练好的模型
model, tokenizer = load_trained_model(args.model_name, args.output_dir)
# 运行推理示例
run_inference_examples(model, tokenizer)
print("\n" + "=" * 80)
print("完成！")
print("=" * 80)
if __name__ == "__main__":
main()
chapter8/MultilingualReasoning/requirements.txt
+21
View File
@@ -0,0 +1,21 @@
# 多语言推理模型微调依赖
# 核心依赖
torch>=2.0.0
transformers>=4.55.0
datasets>=2.14.0
accelerate>=0.20.0
# 微调相关
trl>=0.20.0
peft>=0.17.0
bitsandbytes>=0.41.0
# 工具和监控
trackio
huggingface-hub>=0.16.0
# 可选：实验跟踪
# wandb
# tensorboard
chapter8/README.ar.md
+49
View File
@@ -0,0 +1,49 @@
# الفصل الثامن · مرحلة ما بعد تدريب النموذج
> أربعة أجزاء: التدريب المسبق وMid-training وSFT وRL؛ منهج السياق الطويل وبناء البيانات، وتثبيت البروتوكول بـSFT، وبيئات RL ومكافآته، وكفاءة العينات من الجولة الواحدة إلى الجولات المتعددة.
← [العودة إلى الملف التمهيدي الرئيسي](../docs/ar/README.md) · 📖 [قراءة نص الفصل](../book-ar/chapter8.ar.md)
## كيفية قراءة التجارب
يستخدم النص هياكل آلية قصيرة لشرح تدفق التحكم؛ ويحتوي دليل التجارب على محولات SDK الكاملة والسجلات والاختبارات وأدلة القبول. لا حاجة لقراءة كل ملف سطرًا سطرًا.
- **Starter:** ابدأ بالهدف والأمر الأدنى وشروط القبول؛ وابدأ من [cot-distillation](cot-distillation/);
- **Builder:** تتبّع نقطة الدخول والحلقة الأساسية ومخطط الحالة/الرسائل والأدوات وأداة التحقق.
- **Maintainer:** ثم اقرأ الاختبارات وmanifest الأدلة ومعالجة الأعطال ومسارات التراجع ومحولات المزوّد.
في القراءة الأولى يمكنك تجاوز بيانات الاعتماد وطبقة العرض وتوافق المزوّد؛ عُد إليها عند إعادة إنتاج رقم.
## المشاريع المصاحبة
| التجربة | المشروع | النوع | الوصف |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | تشغيل Q-learning وAgent LLM في بيئة البحث عن الكنز نفسها للتعلم من الخبرة. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | يقطّر أمثلة المعلم إلى prompt للطالب ويقارن الجودة والتكلفة. |
| 7-3، 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | 📖 | التدريب المسبق لنموذج لغة صغير من البداية لفهم عملية التدريب المسبق الكاملة والتقنيات الأساسية. |
| 7-5 | [تابع التدريب المسبق](continued-pretraining/) | ✅ | ينفذ تدريبًا مسبقًا مستمرًا على البيانات الخاصة بالمجال لتحسين أداء النموذج في المجال المستهدف. |
| 7-6 | [السمسم](sesame/) | ✅ | خطاب Sesame CSM SFT: ضبط LoRA لنموذج 1B TTS، والتحكم في التعبير باستخدام العلامات غير اللغوية مثل `<laugh>` و`<sigh>` |
| 7-6 | [أورفيوس](orpheus/) | ✅ | خطاب Orpheus 3B SFT: ضبط LoRA الدقيق لنموذج TTS، واستنساخ الصوت عبر الصوت المرجعي من أجل تناسق الصوت عبر الجمل |
| 7-7 | [استدلال متعدد اللغات](MultilingualReasoning/) | ✅ | يدرب القدرة الاستدلالية للنماذج في بيئات متعددة اللغات، مما يحسن الأداء في المهام متعددة اللغات. |
| 7-9 | [التقطير في المهد](cot-distillation/) | ✅ | يقطر مسارات CoT من النماذج الحدودية مثل Claude عبر OpenRouter؛ تم التحقق من القاعدة وتحويلها إلى بيانات SFT (مصاحب للتجربة 7-9). |
| 7-10 | [أدابت ثينك](AdaptThink/) | 📖 | يعلم نماذج الاستدلال كيفية اختيار وضع الاستدلال الخاص بهم بشكل تكيفي (التفكير مقابل عدم التفكير) بناءً على صعوبة المشكلة. من خلال التحسين المقيد وأخذ العينات المهمة، فإنه يقلل بشكل كبير من تكاليف الاستدلال (45-69%) مع تحسين الدقة. استنادًا إلى نموذج DeepSeek-R1-Distill-Qwen، والذي تم تدريبه باستخدام خوارزمية DAPO. |
| 7-11 | `SFTvsRL/` | 📖 | يقارن بشكل منهجي فعالية الضبط الدقيق الخاضع للإشراف (SFT) والتعلم المعزز (RL) في المهام المختلفة، ويحلل نقاط القوة والضعف وسيناريوهات التطبيق المناسبة لكلتا الطريقتين. |
| 7-12 | [الاستدلال المكاني](SpatialReasoning/) | 📖 | يركز على تدريب قدرة النماذج على التفكير المكاني للتعامل مع المشكلات التي تتضمن العلاقات المكانية مثل الموضع والاتجاه والمسافة. |
| 7-13 | [بسيطVLA-RL](SimpleVLA-RL/) | 📖 | يجمع بين الرؤية واللغة والعمل في التدريب على التعلم المعزز، مما يمكّن النماذج من فهم المدخلات المرئية وتنفيذ الإجراءات المقابلة. |
| 7-14 | [إعادة أداة](retool/) | 📖 | يستخدم حوارًا متعدد المنعطفات وصندوق حماية الشفرة لتعزيز قدرة التفكير الرياضي لنماذج اللغات الكبيرة. ومن خلال عملية التدريب المكونة من مرحلتين SFT وRL، يتعلم النموذج كيفية استخدام بيئة تنفيذ الشفرة للمساعدة في حل المشكلات الرياضية. استنادًا إلى Qwen2.5-32B-Instruct، الذي تم تدريبه على مجموعة بيانات AIME 2024، باستخدام خوارزمية DAPO ووضع الحماية SandboxFusion. |
| 7-15 | `AWorld/` · [تدريب AWorld](AWorld-train/) | 📖 | يدرّب الوكلاء المتجسدين باستخدام إطار AWorld، بما يمكّنهم من أداء مهام معقدة في بيئة افتراضية والتعلم من الخبرة. |
| 7-16 | [RLVP](RLVP/) | 📖 | بحث ما بعد التدريب لـ RLVP - مكافأة النتيجة، ومعاقبة المسار (مصاحب للتجربة 7-16)؛ يوجد رمز التدريب/التقييم الكامل في مستودع الورق المنفصل `19PINE-AI/rlvp`، والذي تحتاج إلى استنساخه بنفسك. |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | Bad-case DPO repair for premature completion on GPU. |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | تدريب SFT مدقّق للاقتباسات الصينية المنحنية الحساسة للنطاق: 1024/256/256 عينة تدريب/حجز/حدود عبر 10 أنواع مقالات و9 لغات برمجة؛ حقق Qwen3-8B دقة exact قدرها 96.9%/97.7% وحفظ المناطق المحمية 100% على GPU. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | تدريب SFT مدقّق للنسخ byte-exact للسلاسل الخاصة: 1024/256/256 عينة؛ حقق Qwen3-8B نسبة 78.9% في مجموعة الحجز و80.1% في الحدود، مع تدقيق tokenizers ‏Qwen3/Qwen2.5/Mistral. |
| — | `verl/` | 📖 | verl هو إطار تعليمي معزز فعال مصمم خصيصًا لتدريب نماذج اللغات الكبيرة RLHF، ويدعم خوارزميات مختلفة مثل PPO وGRPO وDAPO. |
| — | [الحدس](Intuitor/) | ✅ | يدرب قدرة النماذج على التفكير البديهي، مما يمكنها من إصدار أحكام سريعة ومعقولة دون الحاجة إلى سلاسل تفكير مفصلة. |
| — | `tinker-cookbook/` | 📖 | يجمع العديد من النصائح العملية وأفضل الممارسات للتدريب النموذجي. |
## أنواع المشاريع
| الأيقونة | النوع | المعنى |
| :--: | --- | --- |
| ✅ | **مستقل** | شفرة كاملة قابلة للتشغيل في هذا المستودع بعد إعداد مفتاح API |
| 📖 | **دليل إعادة الإنتاج** | وثائق تفصيلية تعتمد على مستودع خارجي يُجلب باستخدام `git clone` |
| 🚧 | **وثيقة التصميم** | وثيقة تصميم وخطة تنفيذ؛ أما الشفرة القابلة للتشغيل فما تزال قيد التطوير |
chapter8/README.en.md
+48
View File
@@ -0,0 +1,48 @@
# Chapter 8 · Model Post-Training
> Four parts—pre-training, Mid-training, SFT, and RL: long-context curricula and data construction, SFT protocol shaping, RL environments and rewards, and sample efficiency from single-turn to multi-turn Agents.
← [Back to main README](../docs/en/README.md) · 📖 [Read chapter text](../book-en/chapter8.md)
## How to Read the Experiments
The prose uses short mechanism skeletons to explain control flow; the experiment directory contains complete SDK adapters, logs, tests, and acceptance evidence. You do not need to read every file line by line.
- **Starter:** Start with the goal, minimum command, and acceptance conditions; begin with [cot-distillation](cot-distillation/);
- **Builder:** Follow the entry point, core loop, state/message schema, tools, and verifier.
- **Maintainer:** Then read tests, evidence manifests, failure handling, rollback paths, and provider adapters.
On a first pass, skip credential loading, presentation code, and provider-compatibility layers; return when reproducing a number.
## Companion Projects
| Exp. | Project | Type | Description |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | Runs Q-learning and an LLM Agent in the same treasure-hunt environment to learn from experience. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | The [retained campaign](../chapter8/prompt-distillation/validation/exp7-8-kimi3-smollm2-20260730/) contains 160/160 training and 80/80 held-out Kimi K3 teacher receipts, a real CUDA-trained SmolLM2-135M-Instruct LoRA checkpoint, and passes all 8 gates; held-out accuracy is 100% for the teacher, 0% for baseline, and 95% for the trained student. |
| 7-3, 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | ✅ | Experiment 7-3's [canonical report](MiniMind-pretrain/validation/runs/exp7-3-training-report-20260731-v1/report.md) retains 49 historical LLM outputs and eight blind judgments. Experiment 7-4's [canonical report](MiniMind-pretrain/validation/runs/exp7-4-training-report-20260731-v1/report.md) retains all 64 historical outputs across eight VLM configurations and images plus eight real image-aware blind judgments. Original VLM SFT ranked highest at 1.9062 and matched QK-Norm+Muon comparisons did not improve, an explicit negative result. Historical checkpoints are not distributed or required for acceptance. |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | [Canonical training report](continued-pretraining/validation/runs/exp7-5-training-report-20260731-v1/report.md) binds the RTX 4090 three-stage output, 15 generations, five blind ARK judgments, source hashes, and current reproduction revisions; final Korean gained 1.7777, English fell 0.8333, and kimchi factual errors remain explicit. Checkpoints are not distributed or required for acceptance. |
| 7-6 | [sesame](sesame/) | ✅ | Sesame CSM tag SFT completed in the [bounded GPU campaign](speech-sft-experiment/): 60 LoRA updates, held-out loss, matched tag/no-tag audio, detector-proxy evaluation, hashes, and retained failures. |
| 7-6 | [orpheus](orpheus/) | ✅ | Orpheus voice-consistency SFT completed in the [bounded GPU campaign](speech-sft-experiment/): 60 LoRA updates, held-out loss, matched base/adapted audio, timbre-proxy evaluation, hashes, and retained failures. |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | 🚧 | The multilingual reasoning SFT implementation exists; repository-retained completion still requires a checkpoint and a before/after benchmark across Chinese and trained languages. |
| 7-9 | [cot-distillation](cot-distillation/) | ✅ | All 24 Kimi K3 teacher cases completed and were rule-filtered; 23 entered SFT. A real CUDA checkpoint and three-arm comparison are retained. The student's 2/24 versus the baseline's 1/24 is nonsignificant (p=1.0) and is reported as a negative result. |
| 7-10 | [AdaptThink](AdaptThink/) | ✅ | The [checkpoint-free training report](AdaptThink/TRAINING_REPORT.md) records public W&B run `wubbn5tj` on 8×H100. At step 300, mean response length fell on all three benchmarks, while AIME mean@16 accuracy declined by 0.42 pp. The run continued through step 410 and then crashed; checkpoints are not distributed, and no independent checkpoint-evaluation receipt was retained. |
| 7-11 | `SFTvsRL/` | 📖 | Systematically compares the effectiveness of Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) on different tasks, analyzing the strengths, weaknesses, and suitable application scenarios of both methods. |
| 7-12 | [SpatialReasoning](SpatialReasoning/) | 📖 | Focuses on training the spatial reasoning ability of models to handle problems involving spatial relationships such as position, direction, and distance. |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) | 📖 | Combines vision, language, and action in reinforcement learning training, enabling models to understand visual input and execute corresponding actions. |
| 7-14 | [retool](retool/) | 📖 | Uses multi-turn dialogue and a code sandbox to enhance the mathematical reasoning ability of large language models. Through a two-stage training process of SFT and RL, the model learns to use a code execution environment to assist in solving mathematical problems. Based on Qwen2.5-32B-Instruct, trained on the AIME 2024 dataset, using the DAPO algorithm and SandboxFusion sandbox. |
| 7-15 | `AWorld/` · [AWorld-train](AWorld-train/) | 📖 | Trains embodied agents based on the AWorld framework, enabling agents to perform complex tasks in a virtual environment and learn from experience. |
| 7-16 | [RLVP](RLVP/) | 📖 | RLVP post-training research — reward the outcome, penalize the path (companion to Experiment 7-16); the full training/evaluation code lives in the separate paper repository `19PINE-AI/rlvp`, which you need to clone yourself. |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | Bad-case DPO repair for premature completion on GPU. |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | Audited scope-sensitive Chinese curved-quote SFT: 1,024/256/256 train/holdout/boundary cases across 10 article types and 9 programming languages; Qwen3-8B GPU run reaches 96.9%/97.7% exact with 100% protected-region preservation. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | Audited byte-exact special-string SFT: 1,024/256/256 train/holdout/boundary cases; Qwen3-8B reaches 78.9% holdout and 80.1% boundary, with Qwen3/Qwen2.5/Mistral tokenizer round-trip audit. |
| — | `verl/` | 📖 | verl is an efficient reinforcement learning framework specifically designed for RLHF training of large language models, supporting various algorithms such as PPO, GRPO, and DAPO. |
| — | [Intuitor](Intuitor/) | ✅ | Trains the intuitive reasoning ability of models, enabling them to make quick, reasonable judgments without requiring detailed chains of thought. |
| — | `tinker-cookbook/` | 📖 | Collects various practical tips and best practices for model training. |
## Project Types
| Icon | Type | Meaning |
| :--: | --- | --- |
| ✅ | **Standalone** | Full code in this repo, runs after configuring API Key |
| 📖 | **Reproduction Guide** | Detailed doc depending on **external repos** to `git clone` |
| 🚧 | **Design Doc** | Architecture/implementation plan only, runnable code still WIP |
chapter8/README.es.md
+51
View File
@@ -0,0 +1,51 @@
# Capítulo 8 · Posentrenamiento de Modelos
> Cuatro partes—pre-entrenamiento, Mid-training, SFT y RL: currículo y datos de contexto largo, protocolos con SFT, entornos y recompensas de RL, y eficiencia de muestra de una a varias rondas.
← [Volver al README principal](../docs/es/README.md) · 📖 [Leer texto del capítulo](../book-es/chapter8.es.md)
Los límites de implementación, código externo y evidencia directa de cada experimento se detallan en el [registro de aceptación](EXPERIMENT_LEDGER.md).
## Cómo leer los experimentos
El texto usa skeletons breves para explicar el flujo de control; el directorio de experimentos contiene adaptadores SDK completos, registros, pruebas y evidencias de aceptación. No hace falta leer cada archivo línea por línea.
- **Starter:** Empieza por el objetivo, el comando mínimo y la aceptación; comienza con [cot-distillation](cot-distillation/);
- **Builder:** Sigue el punto de entrada, el bucle central, el esquema de estado/mensajes, las herramientas y el verificador.
- **Maintainer:** Después revisa pruebas, manifiestos, fallos, rollback y adaptadores de proveedores.
En la primera pasada puedes omitir credenciales, presentación y compatibilidad de proveedores; vuelve al reproducir una cifra.
## Proyectos Complementarios
| Exp. | Proyecto | Tipo | Descripción |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | En el mismo entorno determinista de búsqueda del tesoro se completaron 10.000 partidas de Q-learning, 100 evaluaciones voraces y una primera ejecución oficial con Moonshot `kimi-k3`; la [evidencia de ambos brazos](../chapter1/learning-from-experience/validation/20260730_011704/evidence.json) conserva 17/17 respuestas originales de la API sin *fallback* |
| 7-3 | [MiniMind-pretrain](MiniMind-pretrain/) · `MiniMind-pretrain/minimind/` | 📖 | Documentación complementaria y código externo `bojieli/minimind` fijado a `8bdc5d9…`; el *checkout* no está presente y el entrenamiento no se ejecutó |
| 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) · `MiniMind-pretrain/minimind-v/` | 📖 | Documentación complementaria y código externo `bojieli/minimind-v` fijado a `ead791c…`; el *checkout* no está presente y el entrenamiento no se ejecutó |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | Preentrenamiento continuo sobre datos de un dominio específico para mejorar su rendimiento |
| 7-6 | [sesame](sesame/) · [orpheus](orpheus/) | 🚧 | Dos vías reales de SFT de voz: modelado con etiquetas paralingüísticas y coherencia de timbre entre frases; se requiere el adaptador entrenado, audio y evidencia comparativa para completarlas |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | 🚧 | Implementación de SFT de razonamiento multilingüe; se necesita un checkpoint entrenado y comparaciones antes/después en benchmarks entre idiomas |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | Implementación transversal de generación de prompts/respuestas del profesor, entrenamiento del alumno y comparación calidad-costo; generar ejemplos o prompts no basta para considerarla completa |
| 7-9 | [cot-distillation](cot-distillation/) | 🚧 | Conserva y filtra por reglas CoT reales de Kimi K3; incluye SFT del alumno sin mocks, comparación de tres brazos sobre los mismos problemas, significación pareada y validación de reflexión/retroceso, pero la máquina actual carece de un checkpoint CUDA |
| 7-10 | [documentación de AdaptThink](AdaptThink/) · `AdaptThink-original/` | 📖 | Código de entrenamiento externo de `bojieli/AdaptThink` para que el modelo elija Thinking/NoThinking según la dificultad |
| 7-11 | `SFTvsRL/` | 📖 | GeneralPoints-L/VL de `bojieli/SFTvsRL`: comparación memoria-generalización ID/OOD entre SFT y PPO con el mismo presupuesto |
| 7-12 | [documentación de SpatialReasoning](SpatialReasoning/) · `SFTvsRL/` | 📖 | Entrenamiento V-IRL-L/VL y evaluación OOD entre ciudades/reglas en el mismo *checkout* de `bojieli/SFTvsRL`; no es un repositorio SpatialReasoning independiente |
| 7-13 | [documentación de SimpleVLA-RL](SimpleVLA-RL/) · `SimpleVLA-RL/SimpleVLA-RL/` | 📖 | Repositorio `PRIME-RL/SimpleVLA-RL` y `verl/` integrado fijados; OpenVLA-OFT, LIBERO/RoboTwin, checkpoints, Flash Attention, CUDA/controlador y recursos del simulador aún no forman un bloqueo de dependencias completamente validado |
| 7-14 | [documentación de retool](retool/) · `verl/` · `SandboxFusion/` | 📖 | La receta ReTool procede de `bojieli/verl` y la ejecución de código en tiempo real depende de `bojieli/SandboxFusion`; no existe un repositorio de código independiente llamado `retool` |
| 7-15 | [documentación de AWorld-train](AWorld-train/) · `AWorld/` | 📖 | Sandbox MCP y entrada de entrenamiento de GAIA en `bojieli/AWorld`, con `bojieli/verl` como backend de entrenamiento |
| 7-16 | [documentación de RLVP](RLVP/) · `RLVP/rlvp/` | 📖 | El código completo de entrenamiento/evaluación procede de `19PINE-AI/rlvp` fijado a `1ad30bc…`; el *checkout* no está presente y el entrenamiento no se ejecutó |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | Reparación DPO de bad cases de finalización prematura en GPU |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | SFT auditado de comillas curvas chinas sensible al ámbito: 1.024/256/256 casos de entrenamiento/reserva/borde, 10 géneros y 9 lenguajes; Qwen3-8B alcanza 96,9%/97,7% exacto y 100% de preservación protegida en GPU |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | SFT auditado de copia byte-exacta: 1.024/256/256 casos; Qwen3-8B alcanza 78,9% en reserva y 80,1% en borde, con auditoría de tokenizadores Qwen3/Qwen2.5/Mistral |
| — | `verl/` | 📖 | Marco eficiente de RLHF para LLM compatible con PPO, GRPO, DAPO y otros algoritmos |
| — | [Intuitor](Intuitor/) | ✅ | Entrena razonamiento intuitivo para obtener decisiones plausibles con rapidez sin depender de una cadena de pensamiento detallada |
| — | `tinker-cookbook/` | 📖 | Colección de técnicas prácticas y mejores prácticas para entrenar modelos |
## Tipos de Proyectos
| Icono | Tipo | Significado |
| :--: | --- | --- |
| ✅ | **Autónomo** | Código completo en este repositorio, se ejecuta tras configurar la Clave API |
| 📖 | **Guía de Reproducción** | Documento detallado que depende de **repositorios externos** para realizar `git clone` |
| 🚧 | **En curso** | Existe una implementación, pero el entrenamiento o la evidencia de aceptación requerida por el texto aún no está completa |
chapter8/README.hu.md
+49
View File
@@ -0,0 +1,49 @@
# 8. fejezet · Modell-utóképzés
> Négy rész—pre-tréning, Mid-training, SFT és RL: hosszúkontextus-tanterv és adatépítés, SFT-protokoll, RL-környezet és jutalom, mintahatékonyság egy- és többmenetes ágenseknél.
← [Vissza a magyar főoldalhoz](../docs/hu/README.md) · 📖 [A fejezet olvasása](../book-hu/chapter8.md)
## Hogyan olvassuk a kísérleteket?
A törzsszöveg rövid mechanizmus-skeletonokkal magyarázza a vezérlési folyamatot; a kísérleti könyvtárakban találhatók a teljes SDK-adapterek, naplók, tesztek és átvételi bizonyítékok. Nem kell minden fájlt sorról sorra elolvasni.
- **Starter:** Kezdje a céllal, a minimális paranccsal és az átvételi feltételekkel; induljon innen: [cot-distillation](cot-distillation/);
- **Builder:** Kövesse a belépési pontot, a fő ciklust, az állapot-/üzenetsémát, az eszközöket és az ellenőrzőt.
- **Maintainer:** Végül olvassa el a teszteket, a bizonyíték-manifeszteket, a hibakezelést, a visszaállítási útvonalakat és a provider-adaptereket.
Első olvasáskor átugorható a hitelesítő adatok betöltése, a megjelenítési réteg és a provider-kompatibilitás; a számok reprodukálásakor térjen vissza.
## Kapcsolódó projektek
| Kísérlet | Projekt | Típus | Leírás |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | Azonos kincskereső környezetben hasonlítja össze a Q-learninget és az LLM-alapú tanulást. |
| 7-3 | [MiniMind-pretrain](MiniMind-pretrain/) · `MiniMind-pretrain/minimind/` | 📖 | Egy kis LLM nulláról történő előképzésének folyamatát mutatja be. |
| 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) · `MiniMind-pretrain/minimind-v/` | 📖 | Egy kis látás-nyelvi modell előképzését és SFT-jét ismerteti. |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | Tartományspecifikus adatokon folytatja az előképzést. |
| 7-6 | [sesame](sesame/) · [orpheus](orpheus/) | 🚧 | Két beszéd-SFT útvonalat vizsgál paralingvisztikai címkékhez és mondatok közötti hangszínkonzisztenciához. |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | 🚧 | Több nyelven tanítja a modell következtetési képességét. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | Tanáradatot készít, diákmodellt képez, majd minőséget és költséget hasonlít össze. |
| 7-9 | [cot-distillation](cot-distillation/) | 🚧 | Helyes CoT-nyomvonalakat szűr, és SFT-adattá alakítja őket. |
| 7-10 | [AdaptThink](AdaptThink/) · `AdaptThink-original/` | 📖 | A feladat nehézsége alapján tanítja meg a modellt a Thinking és NoThinking mód közötti választásra. |
| 7-11 | `SFTvsRL/` | 📖 | Azonos költségkeret mellett hasonlítja össze az SFT memorizálását és az RL általánosítását. |
| 7-12 | [SpatialReasoning](SpatialReasoning/) · `SFTvsRL/` | 📖 | Belső és eloszláson kívüli térbeli következtetést tanít és értékel. |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) · `SimpleVLA-RL/SimpleVLA-RL/` | 📖 | A látást, nyelvet és cselekvést megerősítéses tanulásban kapcsolja össze. |
| 7-14 | [retool](retool/) · `verl/` · `SandboxFusion/` | 📖 | Kódértelmező használatára tanít veRL háttérrendszerrel és végrehajtási sandboxszal. |
| 7-15 | [AWorld-train](AWorld-train/) · `AWorld/` | 📖 | AWorld-alapú GAIA-környezetben tanít eszközhasználó ágenst. |
| 7-16 | [RLVP](RLVP/) · `RLVP/rlvp/` | 📖 | Az RLVP-kutatást reprodukálja: jutalmazza az eredményt, és bünteti az útvonalat. |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | Korai befejezési bad case DPO-javítása GPU-n. |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | Auditált, hatókörérzékeny kínai görbe idézőjel-SFT: 1024/256/256 tanító/holdout/perem eset 10 műfajban és 9 programnyelven; a Qwen3-8B 96,9%/97,7% exact és 100% védett-rész megőrzést ért el GPU-n. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | Auditált bájt-pontos speciális karakterlánc-SFT: 1024/256/256 eset; a Qwen3-8B holdout 78,9%, perem 80,1%, Qwen3/Qwen2.5/Mistral tokenizer-audittal. |
| — | `verl/` | 📖 | Hatékony RLHF-keretrendszer PPO, GRPO, DAPO és további algoritmusok számára. |
| — | [Intuitor](Intuitor/) | ✅ | Hosszú gondolatmenet nélkül tanít intuitív következtetést. |
| — | `tinker-cookbook/` | 📖 | Modellképzési receptek és bevált gyakorlatok gyűjteménye. |
## Projekttípusok
| Ikon | Típus | Jelentés |
| :--: | --- | --- |
| ✅ | **Önálló** | A teljes kód a repository-ban található, és az API-kulcsok beállítása után futtatható. |
| 📖 | **Reprodukciós útmutató** | Külső repository szükséges, amelyet külön kell `git clone` paranccsal letölteni. |
| 🚧 | **Folyamatban** | Az implementáció, a tanítás vagy az elfogadási bizonyíték még nem teljes. |
chapter8/README.id.md
+49
View File
@@ -0,0 +1,49 @@
# Bab 8 · Pasca-Pelatihan Model
> Empat bagian—pre-training, Mid-training, SFT, dan RL: kurikulum serta data konteks panjang, pembentukan protokol SFT, environment dan reward RL, serta efisiensi sampel dari single-turn hingga multi-turn.
← [Kembali ke README utama](../docs/id/README.md) · 📖 [Baca bab](../book-id/chapter8.md)
## Cara Membaca Eksperimen
Teks utama memakai skeleton mekanisme singkat untuk menjelaskan alur kontrol; direktori eksperimen berisi adapter SDK lengkap, log, pengujian, dan bukti penerimaan. Anda tidak perlu membaca setiap berkas baris demi baris.
- **Starter:** Mulai dari tujuan, perintah minimum, dan syarat penerimaan; awali dengan [cot-distillation](cot-distillation/);
- **Builder:** Telusuri titik masuk, loop inti, skema status/pesan, alat, dan verifier.
- **Maintainer:** Terakhir, baca pengujian, manifest bukti, penanganan kegagalan, rollback, dan adapter provider.
Pada pembacaan pertama, lewati kredensial, presentasi, dan kompatibilitas provider; kembali saat mereproduksi angka.
## Proyek Pendamping
| Eksperimen | Proyek | Jenis | Deskripsi |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | Membandingkan Q-learning dan LLM pada lingkungan pencarian harta karun yang sama. |
| 7-3 | [MiniMind-pretrain](MiniMind-pretrain/) · `MiniMind-pretrain/minimind/` | 📖 | Mempelajari proses pre-training LLM kecil dari awal. |
| 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) · `MiniMind-pretrain/minimind-v/` | 📖 | Mempelajari pre-training dan SFT vision-language model kecil. |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | Melanjutkan pre-training pada data domain tertentu. |
| 7-6 | [sesame](sesame/) · [orpheus](orpheus/) | 🚧 | Dua jalur SFT suara untuk tag paralinguistik dan konsistensi timbre lintas kalimat. |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | 🚧 | Melatih kemampuan penalaran dalam beberapa bahasa. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | Membangun data guru, melatih siswa, dan membandingkan kualitas serta biaya. |
| 7-9 | [cot-distillation](cot-distillation/) | 🚧 | Menyaring trajectory CoT yang benar dan menyiapkannya sebagai data SFT. |
| 7-10 | [AdaptThink](AdaptThink/) · `AdaptThink-original/` | 📖 | Mengajarkan model memilih mode Thinking atau NoThinking sesuai kesulitan. |
| 7-11 | `SFTvsRL/` | 📖 | Membandingkan memori dan generalisasi SFT dengan RL pada anggaran yang sama. |
| 7-12 | [SpatialReasoning](SpatialReasoning/) · `SFTvsRL/` | 📖 | Melatih serta mengevaluasi spatial reasoning ID dan OOD. |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) · `SimpleVLA-RL/SimpleVLA-RL/` | 📖 | Menggabungkan visi, bahasa, dan tindakan dalam pelatihan RL. |
| 7-14 | [retool](retool/) · `verl/` · `SandboxFusion/` | 📖 | Melatih penggunaan code interpreter dengan backend veRL dan sandbox eksekusi. |
| 7-15 | [AWorld-train](AWorld-train/) · `AWorld/` | 📖 | Melatih Agent menggunakan tool pada lingkungan GAIA berbasis AWorld. |
| 7-16 | [RLVP](RLVP/) · `RLVP/rlvp/` | 📖 | Mereproduksi riset RLVP: memberi reward pada hasil dan penalti pada jalur. |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | Perbaikan DPO bad case penyelesaian prematur di GPU. |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | SFT tanda kutip lengkung Tionghoa berbasis cakupan yang diaudit: 1.024/256/256 data train/holdout/batas, 10 jenis artikel dan 9 bahasa pemrograman; Qwen3-8B mencapai exact 96,9%/97,7% dan preservasi area terlindungi 100% di GPU. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | SFT penyalinan string khusus byte-exact yang diaudit: 1.024/256/256 data; Qwen3-8B mencapai holdout 78,9% dan batas 80,1%, dengan audit tokenizer Qwen3/Qwen2.5/Mistral. |
| — | `verl/` | 📖 | Framework RLHF efisien untuk PPO, GRPO, DAPO, dan algoritme lain. |
| — | [Intuitor](Intuitor/) | ✅ | Melatih penalaran intuitif tanpa chain-of-thought panjang. |
| — | `tinker-cookbook/` | 📖 | Kumpulan resep dan praktik terbaik pelatihan model. |
## Jenis Proyek
| Ikon | Jenis | Arti |
| :--: | --- | --- |
| ✅ | **Mandiri** | Kode lengkap tersedia di repositori dan dapat dijalankan setelah API Key dikonfigurasi. |
| 📖 | **Panduan Reproduksi** | Memerlukan repositori eksternal yang harus di-`git clone`. |
| 🚧 | **Dalam Proses** | Implementasi, pelatihan, atau bukti penerimaan belum lengkap. |
chapter8/README.ja.md
+49
View File
@@ -0,0 +1,49 @@
# 第8章 · モデルのポストトレーニング
> 事前学習、Mid-training、SFT、RL の 4 段階：長文脈カリキュラムとデータ設計、SFT によるプロトコル形成、RL の環境と報酬、シングルターンからマルチターンまでのサンプル効率。
← [メイン README に戻る](../docs/ja/README.md) · 📖 [章の本文を読む](../book-ja/chapter8.ja.md)
## 実験の読み方
本文では短い mechanism skeleton で制御フローを説明し、実験ディレクトリには完全な SDK アダプター、ログ、テスト、受け入れ証拠を置きます。すべてのファイルを一行ずつ読む必要はありません。
- **Starter:** 目的・最小コマンド・受け入れ条件から始め、まず [cot-distillation](cot-distillation/);
- **Builder:** エントリポイント、中心ループ、状態／メッセージ schema、ツール、検証器を追います。
- **Maintainer:** 最後にテスト、証拠 manifest、失敗処理、rollback 経路、provider adapter を読みます。
初読では認証情報、表示層、provider 互換層を飛ばし、数値を再現するときに戻ってください。
## 付随プロジェクト
| 実験 | プロジェクト | 種類 | 説明 |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | 同じ宝探し環境で Q-learning と LLM Agent を実行し、経験から学習する。 |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | 教師の例を学生 prompt に蒸留し、品質とコストを比較する。 |
| 7-3, 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | 📖 | 小型言語モデルをゼロから事前学習し、事前学習の完全なプロセスと主要技術を理解する。 |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | ドメイン固有のデータで継続事前学習を行い、対象ドメインにおけるモデルの性能を向上させる。 |
| 7-6 | [sesame](sesame/) | ✅ | Sesame CSM 音声 SFT：LoRA で 1B TTS モデルを微調整し、`<laugh>`・`<sigh>` などのパラ言語タグで表現を制御 |
| 7-6 | [orpheus](orpheus/) | ✅ | Orpheus 3B 音声 SFT：LoRA で TTS モデルを微調整し、参照音声の連結で文をまたいだ音色の一貫した声の複製を実現 |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | ✅ | 複数の言語環境におけるモデルの推論能力を訓練し、言語横断タスクの性能を向上させる。 |
| 7-9 | [cot-distillation](cot-distillation/) | ✅ | OpenRouter 経由で Claude などの最先端モデルから CoT 軌跡を蒸留し、ルール検証器でフィルタリングして SFT データを生成する（実験 7-9 対応） |
| 7-10 | [AdaptThink](AdaptThink/) | 📖 | 推論モデルに、問題の難易度に基づいて推論モード（Thinking と NoThinking）を適応的に選択させる。制約付き最適化と重要度サンプリングを通じて、精度を向上させながら推論コストを大幅に削減する（45〜69%）。DeepSeek-R1-Distill-Qwen モデルに基づき、DAPO アルゴリズムを用いて訓練する。 |
| 7-11 | `SFTvsRL/` | 📖 | 教師ありファインチューニング（SFT）と強化学習（RL）が異なるタスクで持つ有効性を体系的に比較し、両手法の長所、短所、適した適用シナリオを分析する。 |
| 7-12 | [SpatialReasoning](SpatialReasoning/) | 📖 | 位置、方向、距離などの空間関係を含む問題を処理するため、モデルの空間推論能力の訓練に焦点を当てる。 |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) | 📖 | 強化学習の訓練において視覚、言語、行動を組み合わせ、モデルが視覚入力を理解して対応する行動を実行できるようにする。 |
| 7-14 | [retool](retool/) | 📖 | 複数ターンの対話とコードサンドボックスを用いて、大規模言語モデルの数学的推論能力を強化する。SFT と RL の 2 段階の訓練プロセスを通じて、モデルはコード実行環境を用いて数学問題の解決を支援することを学ぶ。Qwen2.5-32B-Instruct に基づき、AIME 2024 データセットで DAPO アルゴリズムと SandboxFusion サンドボックスを用いて訓練する。 |
| 7-15 | `AWorld/` · [AWorld-train](AWorld-train/) | 📖 | AWorld フレームワークに基づいて身体化された Agent を訓練し、Agent が仮想環境で複雑なタスクを実行し、経験から学習できるようにする。 |
| 7-16 | [RLVP](RLVP/) | 📖 | 結果に報酬を与え、経路にペナルティを課す（RLVP）事後学習の研究（実験 7-16 対応）。完全な訓練・評価コードは独立した論文リポジトリ `19PINE-AI/rlvp` にあり、各自でクローンが必要 |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | GPU 上の早期完了 bad case に対する DPO 修正。 |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | 監査済みスコープ依存中国語曲線引用符 SFT：10 文書ジャンル・9 プログラミング言語で train/holdout/boundary=1024/256/256、Qwen3-8B は exact 96.9%/97.7%、保護領域保持率 100%。 |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | 監査済み byte-exact 特殊文字列コピー SFT：1024/256/256 件、Qwen3-8B は holdout 78.9%、boundary 80.1%、Qwen3/Qwen2.5/Mistral tokenizer 監査付き。 |
| — | `verl/` | 📖 | verl は、大規模言語モデルの RLHF 訓練のために特別に設計された効率的な強化学習フレームワークで、PPO、GRPO、DAPO などのさまざまなアルゴリズムをサポートする。 |
| — | [Intuitor](Intuitor/) | ✅ | モデルの直感的推論能力を訓練し、詳細な思考連鎖を必要とせずに迅速かつ合理的な判断を下せるようにする。 |
| — | `tinker-cookbook/` | 📖 | モデル訓練のためのさまざまな実践的なコツとベストプラクティスを収集する。 |
## プロジェクトの種類
| アイコン | 種類 | 意味 |
| :--: | --- | --- |
| ✅ | **単独実行** | このリポジトリに完全なコードがあり、API キーを設定すれば実行できる |
| 📖 | **再現ガイド** | `git clone` が必要な**外部リポジトリ**に依存する詳細ドキュメント |
| 🚧 | **設計ドキュメント** | アーキテクチャ/実装計画のみで、実行可能なコードは未完成 |
chapter8/README.ko.md
+49
View File
@@ -0,0 +1,49 @@
# 제8장 · 모델 사후 학습
> 사전 학습, Mid-training, SFT, RL의 네 단계: 긴 문맥 커리큘럼과 데이터 구성, SFT 프로토콜 고정, RL 환경과 보상, 단일 턴에서 다중 턴까지의 샘플 효율을 다룹니다.
← [한국어 메인 README로 돌아가기](../docs/ko/README.md) · 📖 [제8장 본문 읽기](../book-ko/chapter8.ko.md)
## 실험 읽는 방법
본문은 짧은 메커니즘 skeleton으로 제어 흐름을 설명하고, 실험 디렉터리에는 완전한 SDK 어댑터·로그·테스트·검수 증거를 둡니다. 모든 파일을 줄 단위로 읽을 필요는 없습니다.
- **Starter:** 목표, 최소 명령, 검수 조건부터 시작하고 다음에서 출발하세요: [cot-distillation](cot-distillation/);
- **Builder:** 진입점, 핵심 루프, 상태/메시지 스키마, 도구와 verifier를 따라갑니다.
- **Maintainer:** 마지막으로 테스트, 증거 manifest, 실패 처리, rollback 경로와 provider adapter를 읽습니다.
첫 읽기에서는 credential, UI, provider 호환 계층을 건너뛰고 수치를 재현할 때 돌아오세요.
## 연계 프로젝트
| 실험 | 프로젝트 | 유형 | 설명 |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | 동일한 결정론적 보물찾기 환경에서 Q-learning 10,000회, 탐욕 정책 평가 100회, 공식 Moonshot `kimi-k3`의 첫 에피소드 실측을 완료했습니다. [두 실험군의 증거](../chapter1/learning-from-experience/validation/20260730_011704/evidence.json)에 원본 API 응답 기록 17/17건을 보존했으며 fallback은 없었습니다. |
| 7-3 | [MiniMind-pretrain](MiniMind-pretrain/) | ✅ | [정식 학습 보고서](MiniMind-pretrain/validation/runs/exp7-3-training-report-20260731-v1/report.md)는 원본 및 QK-Norm+Muon 모델의 사전 학습·SFT·DPO 단계에서 나온 역사적 출력 49개, 익명 ARK 심사 8회, 고정된 소스·데이터·환경 재현 계약을 보존합니다. 역사적 체크포인트는 배포하지 않으며 승인 요건이 아닙니다. |
| 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | ✅ | [정식 학습 보고서](MiniMind-pretrain/validation/runs/exp7-4-training-report-20260731-v1/report.md)는 8개 VLM 구성 × 8개 이미지의 역사적 출력 64개, 실제 이미지 기반 익명 ARK 심사 8회, 고정된 소스·데이터·CLIP·평가 이미지 해시를 보존합니다. 원본 SFT가 1.9062로 가장 높았고 동일 SFT 기반 QK-Norm+Muon 비교는 개선되지 않았다는 부정적 결과도 명시합니다. 역사적 체크포인트는 배포하지 않으며 승인 요건이 아닙니다. |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | [정식 학습 보고서](continued-pretraining/validation/runs/exp7-5-training-report-20260731-v1/report.md)는 RTX 4090 3단계 원시 출력, 15개 생성, 5회 익명 ARK 심사, 소스 해시와 현재 재현 revision을 결합합니다. 최종 한국어는 +1.7777, 영어는 -0.8333이었고 김치 사실 오류도 명시했습니다. 체크포인트는 배포하지 않으며 승인 요건이 아닙니다. |
| 7-6 | [sesame](sesame/) · [orpheus](orpheus/) | 🚧 | 준언어 태그 모델링과 문장 간 음색 일관성을 다루는 두 가지 실제 음성 SFT 트랙입니다. 학습 후 어댑터, 음성 결과물, 수동·자동 비교 증거가 있어야 완료로 봅니다. |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | 🚧 | 다국어 사고 SFT 구현입니다. 학습 체크포인트와 언어 간 벤치마크의 학습 전후 비교가 있어야 완료로 봅니다. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | 교사 모델의 프롬프트·응답 생성, 학생 모델 학습, 품질·비용 비교를 다루는 장 간 연계 구현입니다. 예시 생성이나 프롬프트 메커니즘만으로는 완료로 보지 않습니다. |
| 7-9 | [cot-distillation](cot-distillation/) | 🚧 | 실제 교사 CoT 궤적을 생성하고 규칙으로 필터링했습니다. 학생 모델을 학습하고 수학·코딩 성능 향상과 성찰·되돌아가기·검증 행동을 확인해야 합니다. |
| 7-10 | [AdaptThink 연계 설명](AdaptThink/) · `AdaptThink-original/` | 📖 | 외부 `bojieli/AdaptThink` 학습 코드로, 문제 난이도에 따라 Thinking 또는 NoThinking을 선택하도록 모델을 학습합니다. |
| 7-11 | `SFTvsRL/` | 📖 | `bojieli/SFTvsRL`의 GeneralPoints-L/VL로, 동일한 예산에서 SFT와 PPO의 ID/OOD 기억·일반화 성능을 비교합니다. |
| 7-12 | [SpatialReasoning 연계 설명](SpatialReasoning/) · `SFTvsRL/` | 📖 | 동일한 `bojieli/SFTvsRL` 체크아웃에서 V-IRL-L/VL 학습과 도시 간·규칙 OOD 평가를 수행하며, 별도의 SpatialReasoning 코드 저장소가 아닙니다. |
| 7-13 | [SimpleVLA-RL 연계 설명](SimpleVLA-RL/) · `SimpleVLA-RL/SimpleVLA-RL/` | 📖 | `PRIME-RL/SimpleVLA-RL` 주 저장소와 내부 `verl/`은 고정되어 있습니다. OpenVLA-OFT, LIBERO/RoboTwin, 체크포인트, Flash Attention, CUDA/드라이버, 시뮬레이터 자산을 아우르는 완전한 의존성 잠금 상태는 아직 검증되지 않았습니다. |
| 7-14 | [retool 연계 설명](retool/) · `verl/` · `SandboxFusion/` | 📖 | ReTool 레시피는 `bojieli/verl`에서 가져오며 실시간 코드 실행은 `bojieli/SandboxFusion`에 의존합니다. `retool`이라는 별도의 소스 저장소는 없습니다. |
| 7-15 | [AWorld-train 연계 설명](AWorld-train/) · `AWorld/` | 📖 | `bojieli/AWorld`의 GAIA MCP 샌드박스와 학습 진입점을 사용하며, 학습 백엔드는 `bojieli/verl`입니다. |
| 7-16 | [RLVP 연계 설명](RLVP/) · `RLVP/rlvp/` | 📖 | 전체 학습·평가 코드는 `1ad30bc…`에 고정된 `19PINE-AI/rlvp`에서 가져옵니다. 현재 체크아웃이 없어 학습은 실행하지 않았습니다. |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | GPU에서 조기 완료 bad case를 DPO로 수정합니다. |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | 감사된 범위 민감 중국어 곡선 따옴표 SFT: 10개 문서 유형·9개 프로그래밍 언어에서 train/holdout/boundary 1024/256/256개; Qwen3-8B exact 96.9%/97.7%, 보호 영역 보존 100%. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | 감사된 특수 문자열 byte-exact 복사 SFT: 1024/256/256개; Qwen3-8B holdout 78.9%, boundary 80.1%, Qwen3/Qwen2.5/Mistral tokenizer 감사 포함. |
| — | `verl/` | 📖 | verl은 대규모 언어 모델의 RLHF 학습을 위해 설계된 효율적인 강화 학습 프레임워크로, PPO, GRPO, DAPO 등 여러 알고리즘을 지원합니다. |
| — | [Intuitor](Intuitor/) | ✅ | 모델의 직관적 사고 능력을 학습해 상세한 사고 사슬 없이도 빠르고 합리적인 판단을 내릴 수 있게 합니다. |
| — | `tinker-cookbook/` | 📖 | 모델 학습을 위한 여러 실전 팁과 모범 사례를 모았습니다. |
## 프로젝트 유형
| 아이콘 | 유형 | 의미 |
| :--: | --- | --- |
| ✅ | **독립 실행** | 전체 코드가 이 저장소에 있으며, API 키를 설정하면 실행할 수 있습니다. |
| 📖 | **재현 가이드** | **외부 저장소**를 `git clone`해야 하는 상세 안내 문서입니다. |
| 🚧 | **진행 중** | 구현은 있지만 학습 또는 본문 기준의 검증 증거가 아직 완전하지 않습니다. |
chapter8/README.md
+105
View File
@@ -0,0 +1,105 @@
# 第 8 章 · 模型后训练
> 预训练/Mid-training/SFT/RL 四阶段：长上下文课程与数据构造、SFT 协议固化、RL 环境与奖励、单轮到多轮和样本效率
← [返回主目录](../README.md) · 📖 [读本章正文](../book/chapter8.md)
逐实验的实现、外部源码与直接证据边界见 [验收台账](EXPERIMENT_LEDGER.md)。
## 如何阅读实验
正文用多组 text skeleton 分开说明 SFT masking、GRPO/PPO rollout、工具 token 屏蔽、RLVP 与蒸馏；完整训练框架和 CUDA 适配留在项目中：
- **Starter**：从 [cot-distillation](cot-distillation/) 先跑 2 题采集/验证 smoke，再按 generate_data.py → train_student.py → evaluate_student.py 追踪；
- **Builder**：按 [RLVP](RLVP/)、[SimpleVLA-RL](SimpleVLA-RL/) 的入口追踪 rollout、验证器和奖励字段；
- **Maintainer**：最后检查数据隔离、checkpoint/环境 hash、显存配置、失败轨迹与留出集；不需要首轮读完 verl/。
正文不要求把实验代码当作可复制的 SDK 教程；读者只需先能定位“哪段实现了正文 skeleton、哪段负责证据”。
## 配套项目
| 编号 | 项目 | 类型 | 一句话说明 |
| :--: | --- | :--: | --- |
| 8-1, 8-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | 同一确定性寻宝环境下完成 10,000 局 Q-learning、100 局贪婪评估与官方 Moonshot `kimi-k3` 第一局实测；[双臂证据](../chapter1/learning-from-experience/validation/20260730_011704/evidence.json)保留 17/17 原始 API 回执且零 fallback |
| 8-3 | [MiniMind-pretrain](MiniMind-pretrain/) · `MiniMind-pretrain/minimind/` | ✅ | [规范训练报告](MiniMind-pretrain/validation/runs/exp8-3-training-report-20260731-v1/report.md)绑定原始与 QK-Norm + Muon 两臂在预训练、SFT、DPO 后的 49 份历史输出、8 次匿名 ARK 盲评、源码/数据/环境复现契约与完整 hash；盲评总分 3.6250 对 2.0417（+1.5833，7 胜 1 平），历史 loss 日志缺失的边界明确保留，checkpoint 不随书分发也不作为验收门槛 |
| 8-4 | [MiniMind-pretrain](MiniMind-pretrain/) | ✅ | [规范训练报告](MiniMind-pretrain/validation/runs/exp8-4-training-report-20260731-v1/report.md)保留 8 配置 × 8 图片的 64 份历史输出及 8 次真实图像感知匿名 ARK 评审，固定原版/改进版源码、数据、CLIP 与评估图片哈希；评审中原版 SFT 最高（1.9062），同 SFT 基座的 QK-Norm+Muon 两阶段均未占优。历史 revision/checkpoint 缺失被明确限定，checkpoint 不随书分发也不作为验收门槛 |
| 8-5 | [continued-pretraining](continued-pretraining/) | ✅ | [规范训练报告](continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/report.md)绑定 RTX-4090 三阶段原始输出、15 份生成、5 次匿名 ARK 盲评、源码与当前复现 revision；韩语最终阶段 +1.7777，英语下降 0.8333，泡菜事实错误明确保留，checkpoint 不随书分发也不作为验收门槛 |
| 8-6 | [sesame](sesame/) · [orpheus](orpheus/) | ✅ | [有界本地 GPU 实验](speech-sft-experiment/)已完成两条真实语音 SFT 轨道：各 60 次 LoRA 更新、留出集损失、40 个基线/微调音频、自动代理指标、哈希与失败样例；不据此声称主观音质 |
| 8-7 | [MultilingualReasoning](MultilingualReasoning/) | 🚧 | 多语言思考 SFT 实现；需训练 checkpoint 与跨语言基准前后对照才算完成 |
| 8-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | [正式保留运行](../chapter8/prompt-distillation/validation/exp8-8-kimi3-smollm2-20260730/)包含 160/160 训练与 80/80 留出 Kimi K3 教师回执、真实 CUDA 训练的 SmolLM2-135M-Instruct LoRA checkpoint，并通过 8/8 门禁；留出集教师 100%、基线 0%、训练后学生 95% |
| 8-9 | [cot-distillation](cot-distillation/) | ✅ | 24/24 Kimi K3 教师轨迹均已完成并经规则过滤，23 条进入 SFT；真实 CUDA checkpoint 与同题三臂对照已保留。学生 2/24 对基线 1/24 的提升不显著（p=1.0），作为负结果如实报告 |
| 8-10 | [AdaptThink 配套说明](AdaptThink/) · `AdaptThink-original/` | ✅ | [历史训练报告](AdaptThink/TRAINING_REPORT.md)记录公开 W&B 主运行 `wubbn5tj`：8×H100，step 300 三基准响应长度均显著下降，但 AIME mean@16 下降 0.42 pp；运行继续至 step 410 后崩溃，checkpoint 不随书分发，且未保留独立 checkpoint 评估回执 |
| 8-11 | `SFTvsRL/` | 📖 | `bojieli/SFTvsRL` 的 GeneralPoints-L/VL：同预算 SFT 与 PPO 的 ID/OOD 记忆—泛化对照 |
| 8-12 | [SpatialReasoning 配套说明](SpatialReasoning/) · `SFTvsRL/` | 📖 | 同一 `bojieli/SFTvsRL` checkout 的 V-IRL-L/VL 训练与跨城市/规则 OOD 评估，不是独立 SpatialReasoning 代码仓库 |
| 8-13 | [SimpleVLA-RL 配套说明](SimpleVLA-RL/) · `SimpleVLA-RL/SimpleVLA-RL/` | 📖 | `PRIME-RL/SimpleVLA-RL` 主仓与内嵌 `verl/` 已固定；OpenVLA-OFT、LIBERO/RoboTwin、checkpoint、Flash Attention、CUDA/driver 和 simulator assets 仍未形成经验证的完整依赖锁 |
| 8-14 | [retool 配套说明](retool/) · `verl/` · `SandboxFusion/` | 📖 | ReTool 配方来自 `bojieli/verl`，实时代码执行依赖 `bojieli/SandboxFusion`；不是一个名为 `retool` 的独立源码仓库 |
| 8-15 | [AWorld-train 配套说明](AWorld-train/) · `AWorld/` | 📖 | `bojieli/AWorld` 中的 GAIA MCP 沙盒与训练入口，`bojieli/verl` 为训练后端 |
| 8-16 | [RLVP 配套说明](RLVP/) · `RLVP/rlvp/` | 📖 | 完整训练/评估代码来自固定到 `1ad30bc…` 的 `19PINE-AI/rlvp`；当前 checkout 缺失，训练未运行 |
| 8-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | 从"过早结束" bad case 到 DPO 修复的完整链路：bad case → 偏好对 → 7B+LoRA 单卡训练 → 未完成任务集与已完成任务保留集验证；本地 RTX PRO 6000 已完成训练，固定候选比较中未完成任务集选对率 25.0% → 91.7%，保留集保持 100% |
| 8-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | 中文弯引号作用域 Bad Case：人工审计合成数据 + 显式 Skill 正反规则 → Qwen3-8B bf16 LoRA SFT → 9 种代码语言和 10 种文章体裁回归；[manifest](curly-quote-sft/validation/manifest.json)；RTX PRO 6000 真实训练已完成，1024/256/256（训练/留出/边界），适配后 exact 96.9%/97.7%，保护区保持 100% |
| 8-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | `old_string`/特殊字符串精确复制 Bad Case：未见随机字符串、相似字符串选择和工具 JSON 参数 → Qwen3-8B bf16 LoRA SFT；[manifest](exact-copy-sft/validation/manifest.json)；RTX PRO 6000 真实训练已完成，1024/256/256（训练/留出/边界），byte-exact 基座 37.5%→适配 78.9%，边界 80.1%；另有 Qwen3/Qwen2.5/Mistral tokenizer 审计 |
| — | `verl/` | 📖 | 为 LLM RLHF 设计的高效 RL 框架，支持 PPO/GRPO/DAPO 等 |
| — | [Intuitor](Intuitor/) | ✅ | 训练模型的直觉推理，快速做出合理判断而不依赖详细思考链 |
| — | `tinker-cookbook/` | 📖 | 收集各种模型训练的实用技巧与最佳实践 |
## 外部训练实验复现锚点
下表严格对应正文实验编号。SHA 来自 2026-07-30 当前工作区 checkout，或同日只读上游审计。8-3、8-4、8-5 有各自的 checkpoint-free 历史训练报告验收包；8-10 提供直接链接公开 W&B 的训练报告。固定 revision 属于未来复现说明，不冒充历史训练时的精确 checkout。其余标为未完成的条目仍只完成来源/路径/入口静态核验，**没有启动训练或外部评测**。
| 实验 | 权威上游 → 本地源码路径 | 固定提交 | 已核对入口 |
| :--: | --- | --- | --- |
| 8-3 | [`bojieli/minimind`](https://github.com/bojieli/minimind) → `chapter8/MiniMind-pretrain/minimind` | `8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795` | `trainer/train_pretrain_muon.py` → `trainer/train_full_sft_muon.py` → `trainer/train_dpo.py`；评估 `eval_model.py` |
| 8-4 | [`bojieli/minimind-v`](https://github.com/bojieli/minimind-v) → `chapter8/MiniMind-pretrain/minimind-v` | `ead791c530fa5f9a3549dbfe9e11ec732d18d2e5` | `trainer/train_pretrain_vlm_muon.py` → `trainer/train_sft_vlm_muon.py`；评估 `eval_vlm.py` |
| 8-10 | [`bojieli/AdaptThink`](https://github.com/bojieli/AdaptThink) → `chapter8/AdaptThink-original` | `0033ad172dd53ac64004b763477407014f21b838`（W&B 历史提交 `9e588202…` 的直接子提交；三个入口文件字节一致） | `bash scripts/preprocess_dataset.sh` → `bash scripts/run_adapt_think_1.5b_deepscaler_16k_delta0.05_btz128_lr2e-6.sh` → `bash scripts/run_eval_verl_hf.sh`；训练命名产生 `-fl-`，评估却硬编码 `-fl4096` 且少一层目录，复现时需手工修正路径 |
| 8-11 | [`bojieli/SFTvsRL`](https://github.com/bojieli/SFTvsRL) → `chapter8/SFTvsRL` | `fef0a4a3367260a0934be1e40b01e4021698e023` | GeneralPoints：`bash scripts/gp_training/language_train.sh` / `bash scripts/gp_training/vl_train.sh`；评估在 `scripts/gp_evaluation/*.sh` |
| 8-12 | 同一 [`bojieli/SFTvsRL`](https://github.com/bojieli/SFTvsRL) → `chapter8/SFTvsRL`；说明在 `chapter8/SpatialReasoning` | `fef0a4a3367260a0934be1e40b01e4021698e023` | V-IRL：`bash scripts/virl_training/vl_train.sh`；ID/规则 OOD/视觉 OOD 分别运行 `scripts/virl_evaluation/vl_{indist,rule_ood,visual_ood}_eval.sh` |
| 8-13 | [论文](https://arxiv.org/abs/2509.09674) · [`PRIME-RL/SimpleVLA-RL`](https://github.com/PRIME-RL/SimpleVLA-RL/tree/7c51662df27b586f9e8a1ab35fcf849f2b8852f9) → `chapter8/SimpleVLA-RL/SimpleVLA-RL` | 主仓及内嵌 `verl/`：`7c51662df27b586f9e8a1ab35fcf849f2b8852f9`；外部栈没有作者给出的兼容 SHA，详见[依赖契约](SimpleVLA-RL/README.md#dependency-contract-and-lock-state) | `bash examples/run_openvla_oft_rl_libero.sh`；RoboTwin2 为 `bash examples/run_openvla_oft_rl_twin2.sh`；两者的 `SFT_MODEL_PATH` 仍是占位符 |
| 8-14 | [`bojieli/verl`](https://github.com/bojieli/verl) → `chapter8/verl`；[`bojieli/SandboxFusion`](https://github.com/bojieli/SandboxFusion) → `chapter8/SandboxFusion` | veRL：`1593fc3a8cf894debdc3dece2a23ed739c282789`；SandboxFusion：`4a0d573ebd64c98234c190a9d1d49e4276199a0c` | 启动沙箱 `make run-online`；在 veRL 根目录运行 `bash recipe/retool/run_qwen2-32b_dapo.sh` |
| 8-15 | [`bojieli/AWorld`](https://github.com/bojieli/AWorld) → `chapter8/AWorld`；训练后端 `chapter8/verl` | AWorld：`a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec`；veRL：`1593fc3a8cf894debdc3dece2a23ed739c282789` | `cd chapter8/AWorld/env && bash run-local.sh`；数据准备后在 `train/examples/train_gaia_with_aworld_verl` 运行 `bash run.sh` |
| 8-16 | [`19PINE-AI/rlvp`](https://github.com/19PINE-AI/rlvp) → `chapter8/RLVP/rlvp` | `1ad30bc7e338911fb733739393d92c420f4d8bee` | 规则/credit 测试 → `scripts/phase0_baseline.py` → `scripts/run_all.sh` → `scripts/eval_checkpoint.py`；完整训练需 CUDA |
从仓库根目录获取当前可固定的版本：
```bash
git clone https://github.com/bojieli/AdaptThink.git chapter8/AdaptThink-original && git -C chapter8/AdaptThink-original checkout --detach 0033ad172dd53ac64004b763477407014f21b838
git clone https://github.com/bojieli/SFTvsRL.git chapter8/SFTvsRL && git -C chapter8/SFTvsRL checkout --detach fef0a4a3367260a0934be1e40b01e4021698e023
git clone https://github.com/PRIME-RL/SimpleVLA-RL.git chapter8/SimpleVLA-RL/SimpleVLA-RL && git -C chapter8/SimpleVLA-RL/SimpleVLA-RL checkout --detach 7c51662df27b586f9e8a1ab35fcf849f2b8852f9
git clone https://github.com/bojieli/verl.git chapter8/verl && git -C chapter8/verl checkout --detach 1593fc3a8cf894debdc3dece2a23ed739c282789
git clone https://github.com/bojieli/AWorld.git chapter8/AWorld && git -C chapter8/AWorld checkout --detach a52d61d6d483e66b22ef16970eae5bbf4f4ab2ec
```
以下四个源码目录当前缺失，但不可变版本已经固定。每组命令都显式 fetch、detached checkout，并核对 `rev-parse HEAD`。8-3 的 checkpoint-free 训练报告已按本书训练实验政策验收；对其他实验而言，源码就绪仍不等于实验完成：
```bash
git clone https://github.com/bojieli/minimind.git chapter8/MiniMind-pretrain/minimind
git -C chapter8/MiniMind-pretrain/minimind fetch origin 8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795
git -C chapter8/MiniMind-pretrain/minimind checkout --detach 8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795
git -C chapter8/MiniMind-pretrain/minimind rev-parse HEAD
test "$(git -C chapter8/MiniMind-pretrain/minimind rev-parse HEAD)" = "8bdc5d97d5845a8c1ac2ed56a5b8b4c0d0fb0795"
git clone https://github.com/bojieli/minimind-v.git chapter8/MiniMind-pretrain/minimind-v
git -C chapter8/MiniMind-pretrain/minimind-v fetch origin ead791c530fa5f9a3549dbfe9e11ec732d18d2e5
git -C chapter8/MiniMind-pretrain/minimind-v checkout --detach ead791c530fa5f9a3549dbfe9e11ec732d18d2e5
git -C chapter8/MiniMind-pretrain/minimind-v rev-parse HEAD
test "$(git -C chapter8/MiniMind-pretrain/minimind-v rev-parse HEAD)" = "ead791c530fa5f9a3549dbfe9e11ec732d18d2e5"
git clone https://github.com/19PINE-AI/rlvp.git chapter8/RLVP/rlvp
git -C chapter8/RLVP/rlvp fetch origin 1ad30bc7e338911fb733739393d92c420f4d8bee
git -C chapter8/RLVP/rlvp checkout --detach 1ad30bc7e338911fb733739393d92c420f4d8bee
git -C chapter8/RLVP/rlvp rev-parse HEAD
test "$(git -C chapter8/RLVP/rlvp rev-parse HEAD)" = "1ad30bc7e338911fb733739393d92c420f4d8bee"
git clone https://github.com/bojieli/SandboxFusion.git chapter8/SandboxFusion
git -C chapter8/SandboxFusion fetch origin 4a0d573ebd64c98234c190a9d1d49e4276199a0c
git -C chapter8/SandboxFusion checkout --detach 4a0d573ebd64c98234c190a9d1d49e4276199a0c
git -C chapter8/SandboxFusion rev-parse HEAD
test "$(git -C chapter8/SandboxFusion rev-parse HEAD)" = "4a0d573ebd64c98234c190a9d1d49e4276199a0c"
```
## 项目类型说明
| 图标 | 类型 | 含义 |
| :--: | --- | --- |
| ✅ | **可独立运行** | 本仓库自带完整代码，配置好 API Key 即可运行 |
| 📖 | **复现指南** | 依赖需自行 `git clone` 的**外部仓库**（训练框架、评测基准等） |
| 🚧 | **进行中** | 已有实现，但训练或正文验收证据尚未完整 |
chapter8/README.ru.md
+48
View File
@@ -0,0 +1,48 @@
# Глава 8 · Постобучение модели
> Четыре части — предобучение, Mid-training, SFT и RL: программа длинного контекста и данные, закрепление протокола через SFT, среды и награды RL, эффективность выборки от одного раунда к многим.
← [К оглавлению](../docs/ru/README.md) · 📖 [Читать главу](../book-ru/chapter8.md)
## Как читать эксперименты
В основном тексте короткие скелеты механизмов объясняют поток управления; в каталогах экспериментов находятся полные адаптеры SDK, журналы, тесты и приёмочные доказательства. Читать каждый файл построчно не требуется.
- **Starter:** Начните с цели, минимальной команды и условий приёмки; начните с [cot-distillation](cot-distillation/);
- **Builder:** Проследите точку входа, основной цикл, схему состояния/сообщений, инструменты и проверяющий модуль.
- **Maintainer:** Затем изучите тесты, манифесты доказательств, обработку сбоев, откат и адаптеры провайдеров.
При первом чтении можно пропустить ключи, слой представления и совместимость провайдеров; вернитесь при воспроизведении чисел.
## Сопутствующие проекты
| Эксп. | Проект | Тип | Описание |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | Запускает Q-learning и LLM-агента в одной среде поиска сокровищ для обучения на опыте. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | Дистиллирует примеры учителя в prompt ученика и сравнивает качество и стоимость. |
| 7-3, 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | 📖 | Предобучает малую языковую модель с нуля, чтобы понять полный процесс предобучения и ключевые технологии. |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | Выполняет продолженное предобучение на доменных данных для повышения качества модели в целевой области. |
| 7-6 | [sesame](sesame/) | ✅ | Sesame CSM speech SFT: дообучение 1B TTS-модели методом LoRA, управление выразительностью паралингвистическими тегами `<laugh>`, `<sigh>` |
| 7-6 | [orpheus](orpheus/) | ✅ | Orpheus 3B speech SFT: дообучение TTS-модели методом LoRA, клонирование голоса по референсному аудио с сохранением тембра между фразами |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | ✅ | Обучает рассуждения модели в многоязычной среде, повышая качество на кросс-язычных задачах. |
| 7-9 | [cot-distillation](cot-distillation/) | ✅ | Дистиллирует CoT-траектории из передовых моделей вроде Claude через OpenRouter; проверяет правилами и превращает в данные для SFT (сопровождает эксперимент 7-9). |
| 7-10 | [AdaptThink](AdaptThink/) | 📖 | Учит рассуждающие модели адаптивно выбирать режим рассуждения (Thinking или NoThinking) в зависимости от сложности задачи. Через оптимизацию с ограничениями и importance sampling значительно снижает стоимость рассуждений (45–69%) при росте точности. На базе модели DeepSeek-R1-Distill-Qwen, обучение алгоритмом DAPO. |
| 7-11 | `SFTvsRL/` | 📖 | Систематически сравнивает эффективность обучения с учителем (SFT) и обучения с подкреплением (RL) на разных задачах, анализируя сильные и слабые стороны и подходящие сценарии обоих методов. |
| 7-12 | [SpatialReasoning](SpatialReasoning/) | 📖 | Сфокусирован на обучении пространственных рассуждений модели для задач с пространственными отношениями — положением, направлением, расстоянием. |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) | 📖 | Объединяет зрение, язык и действие в обучении с подкреплением, позволяя моделям понимать визуальный вход и выполнять соответствующие действия. |
| 7-14 | [retool](retool/) | 📖 | Использует многораундовый диалог и песочницу кода для усиления математических рассуждений больших языковых моделей. Через двухэтапное обучение SFT и RL модель учится применять среду исполнения кода при решении математических задач. На базе Qwen2.5-32B-Instruct, обучение на AIME 2024 алгоритмом DAPO и песочницей SandboxFusion. |
| 7-15 | `AWorld/` · [AWorld-train](AWorld-train/) | 📖 | Обучает воплощённых агентов на базе фреймворка AWorld, позволяя им выполнять сложные задачи в виртуальной среде и учиться на опыте. |
| 7-16 | [RLVP](RLVP/) | 📖 | Исследование постобучения по принципу «награждаем результат, штрафуем путь» (RLVP) (сопровождает эксперимент 7-16); полный код обучения/оценки находится в отдельном репозитории статьи `19PINE-AI/rlvp`, его нужно клонировать самостоятельно |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | DPO-исправление bad case преждевременного завершения на GPU. |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | Аудированный scope-sensitive SFT китайских кавычек: 1024/256/256 train/holdout/boundary, 10 типов статей и 9 языков программирования; Qwen3-8B даёт exact 96,9%/97,7% и 100% сохранения защищённых областей на GPU. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | Аудированный byte-exact SFT специальных строк: 1024/256/256 примеров; Qwen3-8B даёт 78,9% holdout и 80,1% boundary, с аудитом токенизаторов Qwen3/Qwen2.5/Mistral. |
| — | `verl/` | 📖 | verl — эффективный фреймворк обучения с подкреплением, специально созданный для RLHF-обучения больших языковых моделей, поддерживает алгоритмы PPO, GRPO, DAPO и др. |
| — | [Intuitor](Intuitor/) | ✅ | Обучает интуитивные рассуждения модели, позволяя ей делать быстрые обоснованные суждения без развёрнутой цепочки рассуждений. |
| — | `tinker-cookbook/` | 📖 | Собирает практические приёмы и лучшие практики обучения моделей. |
## Типы проектов
| Значок | Тип | Значение |
| :--: | --- | --- |
| ✅ | **Автономный** | Полный код в этом репозитории, запускается после настройки API-ключа |
| 📖 | **Гайд по воспроизведению** | Подробный документ, зависящий от **внешних репозиториев** через `git clone` |
| 🚧 | **Проектный документ** | Только архитектура/план реализации, рабочий код ещё в разработке |
chapter8/README.ta.md
+49
View File
@@ -0,0 +1,49 @@
# அத்தியாயம் 8 · மாதிரி பிந்தைய பயிற்சி
> முன்-பயிற்சி, Mid-training, SFT, RL என்ற நான்கு பகுதிகள்: நீண்ட சூழல் curriculum மற்றும் தரவு அமைப்பு, SFT protocol, RL சூழல் மற்றும் reward, ஒற்றைச் சுற்றிலிருந்து பல சுற்று வரை sample efficiency.
← [முக்கிய README க்குத் திரும்பு](../docs/ta/README.md) · 📖 [அத்தியாய உரையைப் படி](../book-ta/chapter8.ta.md)
## சோதனைகளை எப்படிப் படிப்பது
முதன்மை உரை குறுகிய mechanism skeleton-களால் control flow-ஐ விளக்குகிறது; முழு SDK adapters, logs, tests, acceptance evidence ஆகியவை experiment கோப்பகத்தில் உள்ளன. ஒவ்வொரு கோப்பையும் வரி வரியாகப் படிக்க வேண்டியதில்லை.
- **Starter:** இலக்கு, குறைந்தபட்ச கட்டளை, ஏற்றுக்கொள்ளும் நிபந்தனைகளில் தொடங்குங்கள்; முதலில் [cot-distillation](cot-distillation/);
- **Builder:** நுழைவுப் புள்ளி, மையச் சுழற்சி, state/message schema, கருவிகள், verifier ஆகியவற்றைப் பின்தொடருங்கள்.
- **Maintainer:** பின்னர் tests, evidence manifest, தோல்வி கையாளல், rollback பாதை, provider adapter ஆகியவற்றைப் படியுங்கள்.
முதல் வாசிப்பில் credentials, UI, provider-compatibility அடுக்குகளைத் தவிர்க்கலாம்; முடிவுகளை மீண்டும் உருவாக்கும்போது திரும்பிப் பாருங்கள்.
## துணை திட்டங்கள்
| சோதனை | Project | Type | Description |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | அனுபவத்திலிருந்து கற்க ஒரே treasure-hunt சூழலில் Q-learning மற்றும் LLM Agent-ஐ இயக்குகிறது. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | ஆசிரியர் எடுத்துக்காட்டுகளை மாணவர் prompt-ஆக distill செய்து தரம் மற்றும் செலவை ஒப்பிடுகிறது. |
| 7-3, 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | 📖 | பூஜ்ஜியத்திலிருந்து சிறிய மொழி மாதிரிகளை முன்-பயிற்றுவித்து, முன்-பயிற்சியின் முழுமையான செயல்முறையையும் முக்கிய தொழில்நுட்பங்களையும் புரிந்துகொள்ளுங்கள். |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | குறிப்பிட்ட களத் தரவுகளில் தொடர்ச்சியான முன்-பயிற்சியை மேற்கொண்டு, இலக்குக் களத்தில் மாதிரியின் செயல்திறனை மேம்படுத்துகிறது. |
| 7-6 | [sesame](sesame/) | ✅ | Sesame CSM குரல் SFT: 1B TTS மாதிரியை LoRA முறையில் நுண்சரிவு செய்து, `<laugh>`, `<sigh>` போன்ற துணைமொழிக் குறிச்சொற்களால் வெளிப்பாட்டைக் கட்டுப்படுத்துதல் |
| 7-6 | [orpheus](orpheus/) | ✅ | Orpheus 3B குரல் SFT: TTS மாதிரியை LoRA முறையில் நுண்சரிவு செய்து, குறிப்பு ஒலியை இணைத்து வாக்கியங்களுக்கு இடையே ஒரே குரல் ஒலித்தன்மையுடன் குரல் நகலெடுப்பு |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | ✅ | பல மொழிச் சூழல்களில் மாதிரியின் பகுத்தறிவுத் திறனைப் பயிற்றுவித்து, குறுக்கு-மொழிப் பணிகளில் செயல்திறனை மேம்படுத்துகிறது. |
| 7-9 | [cot-distillation](cot-distillation/) | ✅ | OpenRouter வழியாக Claude போன்ற முன்னணி மாதிரிகளிலிருந்து CoT பாதைகளை திரட்டி, விதி சரிபார்ப்பாளரால் வடிகட்டி SFT தரவை உருவாக்குகிறது (சோதனை 7-9 துணை). |
| 7-10 | [AdaptThink](AdaptThink/) | 📖 | பகுத்தறிவு மாதிரிகள் கேள்வியின் சிரமத்திற்கு ஏற்ப பகுத்தறிவு முறையை (Thinking vs NoThinking) தகவமைப்புடன் தேர்ந்தெடுக்கக் கற்றுக்கொடுக்கிறது. கட்டுப்படுத்தப்பட்ட உகப்பாக்கம் மற்றும் முக்கியத்துவ மாதிரியெடுப்பு மூலம், பகுத்தறிவு செலவை (45-69%) பெரிதும் குறைத்த அதே வேளையில் துல்லியத்தையும் மேம்படுத்துகிறது. DeepSeek-R1-Distill-Qwen மாதிரியை அடிப்படையாகக் கொண்டு, DAPO அல்காரிதம் மூலம் பயிற்றுவிக்கப்பட்டது. |
| 7-11 | `SFTvsRL/` | 📖 | வெவ்வேறு பணிகளில் மேற்பார்வையிடப்பட்ட நுண்-சரிப்படுத்தல் (SFT) மற்றும் வலுவூட்டல் கற்றல் (RL) ஆகியவற்றின் விளைவுகளை முறையாக ஒப்பிட்டு, இரண்டு முறைகளின் நன்மை தீமைகளையும் பொருந்தும் சூழ்நிலைகளையும் பகுப்பாய்வு செய்கிறது. |
| 7-12 | [SpatialReasoning](SpatialReasoning/) | 📖 | இருப்பிடம், திசை, தூரம் போன்ற இடவியல் உறவுகளை உள்ளடக்கிய கேள்விகளைக் கையாள்வதற்காக மாதிரியின் இடவியல் பகுத்தறிவுத் திறனைப் பயிற்றுவிப்பதில் கவனம் செலுத்துகிறது. |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) | 📖 | பார்வை, மொழி மற்றும் செயலை இணைக்கும் வலுவூட்டல் கற்றல் பயிற்சி, பார்வை உள்ளீடுகளைப் புரிந்துகொண்டு தொடர்புடைய செயல்களைச் செயல்படுத்த மாதிரியைச் செய்கிறது. |
| 7-14 | [retool](retool/) | 📖 | பல-சுற்று உரையாடல்கள் மற்றும் குறியீடு மணல் பெட்டியைப் பயன்படுத்தி பெரிய மொழி மாதிரிகளின் கணிதப் பகுத்தறிவுத் திறனை மேம்படுத்துகிறது. SFT மற்றும் RL என்ற இரண்டு-கட்டப் பயிற்சி மூலம், கணிதக் கேள்விகளைத் தீர்ப்பதற்கு உதவும் வகையில் குறியீடு செயல்படுத்தும் சூழலைப் பயன்படுத்த மாதிரி கற்றுக்கொள்கிறது. Qwen2.5-32B-Instruct ஐ அடிப்படையாகக் கொண்டு, AIME 2024 தரவுத்தொகுப்பில் பயிற்றுவிக்கப்பட்டது, DAPO அல்காரிதம் மற்றும் SandboxFusion மணல் பெட்டியைப் பயன்படுத்துகிறது. |
| 7-15 | `AWorld/` · [AWorld-train](AWorld-train/) | 📖 | AWorld கட்டமைப்பின் அடிப்படையில் உடல்சார் ஏஜென்ட்டுகளைப் பயிற்றுவித்து, ஏஜென்ட்டுகள் மெய்நிகர் சூழலில் சிக்கலான பணிகளைச் செயல்படுத்தவும் அனுபவத்திலிருந்து கற்றுக்கொள்ளவும் செய்கிறது. |
| 7-16 | [RLVP](RLVP/) | 📖 | முடிவுக்கு வெகுமதியும் பாதைக்கு தண்டனையும் (RLVP) அளிக்கும் பிந்தைய பயிற்சி ஆராய்ச்சி (சோதனை 7-16 துணை); முழுமையான பயிற்சி/மதிப்பீட்டுக் குறியீடு தனித்த ஆய்வுக் கட்டுரை களஞ்சியமான `19PINE-AI/rlvp`-இல் உள்ளது, நீங்களே clone செய்ய வேண்டும் |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | GPU மூலம் premature-completion bad case DPO திருத்தம். |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | தணிக்கை செய்யப்பட்ட scope-sensitive சீன வளைந்த மேற்கோள் SFT: 10 கட்டுரை வகைகள், 9 நிரலாக்க மொழிகளில் train/holdout/boundary 1024/256/256; Qwen3-8B exact 96.9%/97.7%, பாதுகாக்கப்பட்ட பகுதி 100%. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | தணிக்கை செய்யப்பட்ட byte-exact சிறப்பு சரம் நகல் SFT: 1024/256/256 மாதிரிகள்; Qwen3-8B holdout 78.9%, boundary 80.1%, Qwen3/Qwen2.5/Mistral tokenizer தணிக்கையுடன். |
| — | `verl/` | 📖 | verl என்பது பெரிய மொழி மாதிரி RLHF பயிற்சிக்காகப் பிரத்யேகமாக வடிவமைக்கப்பட்ட திறமையான வலுவூட்டல் கற்றல் கட்டமைப்பாகும், PPO, GRPO, DAPO உட்படப் பல அல்காரிதங்களை ஆதரிக்கிறது. |
| — | [Intuitor](Intuitor/) | ✅ | மாதிரியின் உள்ளுணர்வுப் பகுத்தறிவுத் திறனைப் பயிற்றுவித்து, விரிவான சிந்தனைச் சங்கிலி தேவையில்லாமலேயே மாதிரி விரைவாக நியாயமான தீர்ப்புகளை வழங்கச் செய்கிறது. |
| — | `tinker-cookbook/` | 📖 | பல்வேறு மாதிரி பயிற்சிக்கான நடைமுறை நுட்பங்கள் மற்றும் சிறந்த நடைமுறைகளைத் தொகுக்கிறது. |
## திட்ட வகைகள்
| சின்னம் | வகை | பொருள் |
| :--: | --- | --- |
| ✅ | **தனித்து இயங்கும்** | முழு குறியீடு இந்த களஞ்சியத்தில், API Key உள்ளமைத்தவுடன் இயங்கும் |
| 📖 | **மறு உருவாக்க வழிகாட்டி** | **வெளிப்புற களஞ்சியங்களை** `git clone` செய்ய வேண்டிய விரிவான ஆவணம் |
| 🚧 | **வடிவமைப்பு ஆவணம்** | கட்டமைப்பு/செயலாக்கத் திட்டம் மட்டும், இயங்கும் குறியீடு இன்னும் WIP |
chapter8/README.tr.md
+49
View File
@@ -0,0 +1,49 @@
# Bölüm 8 · Model Eğitim Sonrası
> Dört bölüm—pre-training, Mid-training, SFT ve RL: uzun bağlam müfredatı ve veri yapımı, SFT ile protokol, RL ortamı ve ödülü, tek turdan çok tura örneklem verimliliği.
← [Ana README'ye dön](../README.tr.md) · 📖 [Bölüm metnini oku](../book-tr/chapter8.tr.md)
## Deneyler nasıl okunur
Metin, kontrol akışını açıklamak için kısa mekanizma skeleton'ları kullanır; deney dizininde tam SDK adaptörleri, günlükler, testler ve kabul kanıtı bulunur. Her dosyayı satır satır okumanız gerekmez.
- **Starter:** Hedef, en kısa komut ve kabul koşullarıyla başlayın; önce [cot-distillation](cot-distillation/);
- **Builder:** Giriş noktasını, ana döngüyü, durum/mesaj şemasını, araçları ve doğrulayıcıyı izleyin.
- **Maintainer:** Son olarak testleri, kanıt manifestlerini, hata işlemeyi, rollback yollarını ve sağlayıcı adaptörlerini okuyun.
İlk okumada kimlik bilgisi yükleme, sunum katmanı ve sağlayıcı uyumluluğunu atlayıp sayıları yeniden üretirken dönün.
## Eşlik Eden Projeler
| Proje | Tür | Açıklama |
| --- | :--: | --- |
| [learning-from-experience](../chapter1/learning-from-experience/) (7-1, 7-2) | ✅ | Deneyimden öğrenmek için aynı hazine avı ortamında Q-learning ve LLM Agent çalıştırır. |
| [prompt-distillation](../chapter8/prompt-distillation/) (7-8) | ✅ | Öğretmen örneklerini öğrenci prompt'una damıtır ve kalite ile maliyeti karşılaştırır. |
| [AdaptThink](AdaptThink/) | 📖 | Muhakeme modellerine, problem zorluğuna göre muhakeme modunu (Thinking vs NoThinking) uyarlanabilir şekilde seçmeyi öğretir. Kısıtlı optimizasyon ve önem örneklemesi yoluyla, doğruluğu artırırken muhakeme maliyetlerini önemli ölçüde azaltır (%45-69). DeepSeek-R1-Distill-Qwen modeline dayanır, DAPO algoritmasıyla eğitilir. |
| [retool](retool/) | 📖 | Büyük dil modellerinin matematiksel muhakeme yeteneğini artırmak için çok turlu diyalog ve bir kod sandbox'ı kullanır. SFT ve RL'den oluşan iki aşamalı bir eğitim süreciyle model, matematik problemlerini çözmeye yardımcı olmak için bir kod yürütme ortamını kullanmayı öğrenir. Qwen2.5-32B-Instruct'a dayanır, AIME 2024 veri kümesinde DAPO algoritması ve SandboxFusion sandbox'ı kullanılarak eğitilir. |
| `AWorld/` · [AWorld-train](AWorld-train/) | 📖 | AWorld çerçevesine dayalı olarak somutlaşmış (embodied) ajanları eğitir; ajanların sanal bir ortamda karmaşık görevleri yerine getirmesini ve deneyimden öğrenmesini sağlar. |
| `SFTvsRL/` | 📖 | Denetimli İnce Ayar (SFT) ve Pekiştirmeli Öğrenmenin (RL) farklı görevlerdeki etkinliğini sistematik olarak karşılaştırır; her iki yöntemin güçlü yanlarını, zayıf yanlarını ve uygun uygulama senaryolarını analiz eder. |
| [premature-completion-dpo](premature-completion-dpo/) (7-17) | ✅ | GPU üzerinde erken tamamlama bad case için DPO düzeltmesi. |
| [curly-quote-sft](curly-quote-sft/) (7-18) | ✅ | Denetimli kapsam duyarlı Çince kıvrımlı tırnak SFT'si: 10 belge türü ve 9 programlama dilinde 1024/256/256 train/holdout/sınır örneği; Qwen3-8B exact 96,9%/97,7%, korunan alan 100%. |
| [exact-copy-sft](exact-copy-sft/) (7-19) | ✅ | Denetimli byte-exact özel dize kopyalama SFT'si: 1024/256/256 örnek; Qwen3-8B holdout 78,9%, sınır 80,1%, Qwen3/Qwen2.5/Mistral tokenizer denetimiyle. |
| `verl/` | 📖 | verl, büyük dil modellerinin RLHF eğitimi için özel olarak tasarlanmış verimli bir pekiştirmeli öğrenme çerçevesidir; PPO, GRPO ve DAPO gibi çeşitli algoritmaları destekler. |
| [Intuitor](Intuitor/) | ✅ | Modellerin sezgisel muhakeme yeteneğini eğitir; ayrıntılı düşünce zincirleri gerektirmeden hızlı, makul kararlar vermelerini sağlar. |
| [MultilingualReasoning](MultilingualReasoning/) | ✅ | Modellerin çok dilli ortamlardaki muhakeme yeteneğini eğitir; diller arası görevlerdeki performansı artırır. |
| [cot-distillation](cot-distillation/) | ✅ | Claude gibi öncü modellerden OpenRouter aracılığıyla CoT trajectory'leri damıtır, bunları kural tabanlı doğrulayıcılarla süzer ve Deney 7-9 için SFT verisi üretir. |
| [SpatialReasoning](SpatialReasoning/) | 📖 | Konum, yön ve mesafe gibi uzamsal ilişkileri içeren problemleri ele almak için modellerin uzamsal muhakeme yeteneğini eğitmeye odaklanır. |
| [SimpleVLA-RL](SimpleVLA-RL/) | 📖 | Görsel, dil ve eylemi pekiştirmeli öğrenme eğitiminde birleştirir; modellerin görsel girdiyi anlamasını ve karşılık gelen eylemleri yürütmesini sağlar. |
| [RLVP](RLVP/) | 📖 | Sonucu ödüllendirip hatalı yolu cezalandıran RLVP sonradan eğitim araştırmasıdır; tam eğitim ve değerlendirme kodu harici \`19PINE-AI/rlvp\` deposundan klonlanır. |
| [continued-pretraining](continued-pretraining/) | ✅ | Hedef alandaki model performansını artırmak için alana özgü veriler üzerinde sürekli ön eğitim yapar. |
| [MiniMind-pretrain](MiniMind-pretrain/) | 📖 | Tam ön eğitim sürecini ve temel teknikleri anlamak için küçük bir dil modelini sıfırdan ön eğitir. |
| [sesame](sesame/) | ✅ | Dizi modelleme görevleri için eğitim ve değerlendirme yöntemlerine odaklanır. |
| [orpheus](orpheus/) | ✅ | Müzik üretimi ve anlama için modeller eğitir. |
| `tinker-cookbook/` | 📖 | Model eğitimi için çeşitli pratik ipuçları ve en iyi uygulamaları bir araya toplar. |
## Proje Türleri
| İkon | Tür | Anlamı |
| :--: | --- | --- |
| ✅ | **Bağımsız** | Bu depoda tam kod, API Key yapılandırıldıktan sonra çalışır |
| 📖 | **Yeniden Üretim Rehberi** | `git clone` ile **harici depolara** bağımlı ayrıntılı belge |
| 🚧 | **Tasarım Belgesi** | Yalnızca mimari/uygulama planı, çalıştırılabilir kod henüz hazır değil |
chapter8/README.vi.md
+49
View File
@@ -0,0 +1,49 @@
# Chương 8 · Hậu huấn luyện mô hình
> Bốn phần—tiền huấn luyện, Mid-training, SFT và RL: curriculum và dữ liệu ngữ cảnh dài, định hình giao thức bằng SFT, môi trường và phần thưởng RL, cùng hiệu quả mẫu từ một vòng đến nhiều vòng.
← [Về README chính](../docs/vi/README.md) · 📖 [Đọc nội dung chương](../book-vi/chapter8.vi.md)
## Cách đọc các thí nghiệm
Phần văn bản dùng skeleton cơ chế ngắn để giải thích luồng điều khiển; thư mục thí nghiệm chứa adapter SDK đầy đủ, log, kiểm thử và bằng chứng nghiệm thu. Không cần đọc từng tệp theo từng dòng.
- **Starter:** Bắt đầu từ mục tiêu, lệnh tối thiểu và điều kiện nghiệm thu; hãy bắt đầu với [cot-distillation](cot-distillation/);
- **Builder:** Lần theo điểm vào, vòng lặp lõi, schema trạng thái/tin nhắn, công cụ và verifier.
- **Maintainer:** Sau đó đọc test, manifest bằng chứng, xử lý lỗi, đường rollback và adapter nhà cung cấp.
Lần đầu có thể bỏ qua credential, lớp trình bày và tương thích provider; quay lại khi cần tái tạo số liệu.
## Dự án đi kèm
| Thí nghiệm | Project | Type | Description |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | Chạy Q-learning và LLM Agent trong cùng môi trường săn kho báu để học từ kinh nghiệm. |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | Chưng cất ví dụ của giáo viên thành prompt học viên và so sánh chất lượng với chi phí. |
| 7-3, 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | 📖 | Tiền huấn luyện mô hình ngôn ngữ nhỏ từ con số 0, hiểu toàn bộ quy trình và kỹ thuật then chốt của tiền huấn luyện. |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | Tiếp tục tiền huấn luyện trên dữ liệu miền cụ thể để nâng cao biểu hiện của mô hình trong miền mục tiêu. |
| 7-6 | [sesame](sesame/) | ✅ | Sesame CSM speech SFT: tinh chỉnh LoRA mô hình TTS 1B, điều khiển biểu cảm bằng các thẻ cận ngôn ngữ như `<laugh>`, `<sigh>` |
| 7-6 | [orpheus](orpheus/) | ✅ | Orpheus 3B speech SFT: tinh chỉnh LoRA mô hình TTS, nhân bản giọng nói qua âm thanh tham chiếu để giữ chất giọng nhất quán xuyên câu |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | ✅ | Huấn luyện năng lực suy luận của mô hình trong môi trường nhiều ngôn ngữ, nâng cao biểu hiện trên các nhiệm vụ xuyên ngôn ngữ. |
| 7-9 | [cot-distillation](cot-distillation/) | ✅ | Chưng cất quỹ đạo CoT từ các mô hình tiên phong như Claude qua OpenRouter; lọc bằng bộ kiểm chứng luật để tạo dữ liệu SFT (kèm Thí nghiệm 7-9). |
| 7-10 | [AdaptThink](AdaptThink/) | 📖 | Cho mô hình suy luận học cách chọn chế độ suy luận thích ứng theo độ khó của câu hỏi (Thinking vs NoThinking). Thông qua tối ưu có ràng buộc và importance sampling, dự án giảm mạnh chi phí suy luận (45–69%) đồng thời nâng cao độ chính xác. Dựa trên mô hình DeepSeek-R1-Distill-Qwen, huấn luyện bằng thuật toán DAPO. |
| 7-11 | `SFTvsRL/` | 📖 | So sánh có hệ thống hiệu quả của fine-tuning có giám sát (SFT) và học tăng cường (RL) trên các nhiệm vụ khác nhau, phân tích ưu nhược điểm và ngữ cảnh phù hợp của hai phương pháp. |
| 7-12 | [SpatialReasoning](SpatialReasoning/) | 📖 | Tập trung huấn luyện năng lực suy luận không gian của mô hình, xử lý các vấn đề liên quan đến vị trí, phương hướng, khoảng cách và các quan hệ không gian khác. |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) | 📖 | Huấn luyện học tăng cường kết hợp thị giác, ngôn ngữ và hành động, giúp mô hình hiểu đầu vào thị giác và thực hiện hành động tương ứng. |
| 7-14 | [retool](retool/) | 📖 | Dùng hội thoại nhiều vòng và sandbox mã để nâng cao năng lực suy luận toán học của mô hình ngôn ngữ lớn. Thông qua hai giai đoạn SFT và RL, mô hình học cách dùng môi trường thực thi mã để hỗ trợ giải bài toán. Dựa trên Qwen2.5-32B-Instruct, huấn luyện trên bộ AIME 2024, dùng thuật toán DAPO và sandbox SandboxFusion. |
| 7-15 | `AWorld/` · [AWorld-train](AWorld-train/) | 📖 | Huấn luyện Agent hiện thân dựa trên framework AWorld, giúp Agent thực thi nhiệm vụ phức tạp trong môi trường ảo và học từ kinh nghiệm. |
| 7-16 | [RLVP](RLVP/) | 📖 | Nghiên cứu hậu huấn luyện RLVP (thưởng cho kết quả, phạt đường đi), dự án đi kèm Thí nghiệm 7-16; mã huấn luyện/đánh giá đầy đủ nằm trong kho bài báo riêng `19PINE-AI/rlvp`, cần tự clone. |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | Sửa bad case hoàn thành sớm bằng DPO trên GPU. |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | SFT dấu ngoặc kép cong tiếng Trung theo phạm vi đã kiểm toán: 1024/256/256 mẫu train/holdout/biên, 10 thể loại và 9 ngôn ngữ lập trình; Qwen3-8B đạt exact 96,9%/97,7%, bảo toàn vùng bảo vệ 100%. |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | SFT sao chép chuỗi đặc biệt byte-exact đã kiểm toán: 1024/256/256 mẫu; Qwen3-8B đạt holdout 78,9%, biên 80,1%, kèm kiểm toán tokenizer Qwen3/Qwen2.5/Mistral. |
| — | `verl/` | 📖 | verl là framework học tăng cường hiệu quả được thiết kế riêng cho huấn luyện RLHF của mô hình ngôn ngữ lớn, hỗ trợ nhiều thuật toán như PPO, GRPO, DAPO. |
| — | [Intuitor](Intuitor/) | ✅ | Huấn luyện năng lực suy luận trực giác của mô hình, giúp mô hình có thể nhanh chóng đưa ra phán đoán hợp lý mà không cần chuỗi suy nghĩ chi tiết. |
| — | `tinker-cookbook/` | 📖 | Tập hợp nhiều kỹ thuật thực dụng và best practice cho huấn luyện mô hình. |
## Phân loại dự án
| Biểu tượng | Loại | Ý nghĩa |
| :--: | --- | --- |
| ✅ | **Chạy độc lập** | Có mã đầy đủ trong kho, chạy được sau khi cấu hình API Key |
| 📖 | **Hướng dẫn tái hiện** | Tài liệu chi tiết, cần `git clone` **kho ngoài** |
| 🚧 | **Tài liệu thiết kế** | Chỉ có kiến trúc/phương án, mã chạy được đang hoàn thiện |
chapter8/README.zh-TW.md
+49
View File
@@ -0,0 +1,49 @@
# 第 8 章 · 模型後訓練
> 預訓練／Mid-training／SFT／RL 四階段：長上下文課程與資料構造、SFT 協定固化、RL 環境與獎勵，以及從單輪到多輪的樣本效率。
← [返回主目錄](../docs/zh-TW/README.md) · 📖 [讀本章正文](../book/chapter8.md)
## 如何閱讀實驗
正文用短小的機制 skeleton 說明控制流；實驗目錄放完整的 SDK 適配、日誌、測試與驗收證據，不需要逐行讀完每個檔案。
- **Starter:** 先讀目標、最小指令與驗收條件；可從 [cot-distillation](cot-distillation/);
- **Builder:** 沿著入口、核心迴圈、狀態／訊息 schema、工具與驗證器閱讀。
- **Maintainer:** 最後再看測試、證據 manifest、失敗處理、回滾路徑與 provider adapter。
第一次閱讀可先跳過憑證載入、展示層和 provider 相容層；要重現數字時再回來查看。
## 配套專案
| 編號 | 專案 | 型別 | 一句話說明 |
| :--: | --- | :--: | --- |
| 7-1, 7-2 | [learning-from-experience](../chapter1/learning-from-experience/) | ✅ | 在同一尋寶環境執行 Q-learning 與 LLM Agent，從經驗中學習。 |
| 7-8 | [prompt-distillation](../chapter8/prompt-distillation/) | ✅ | 將教師範例蒸餾為學生 prompt，並比較品質與成本。 |
| 7-3, 7-4 | [MiniMind-pretrain](MiniMind-pretrain/) | 📖 | 從零預訓練小型 LLM/VLM，理解完整預訓練流程與關鍵技術 |
| 7-5 | [continued-pretraining](continued-pretraining/) | ✅ | 在特定領域資料上持續預訓練，提升目標領域表現 |
| 7-6 | [sesame](sesame/) | ✅ | Sesame CSM 語音 SFT：LoRA 微調 1B TTS 模型，用 `<laugh>`、`<sigh>` 等副語言標記控制表達 |
| 7-6 | [orpheus](orpheus/) | ✅ | Orpheus 3B 語音 SFT：LoRA 微調 TTS 模型，拼接參考音訊實現跨句音色一致的聲音複刻 |
| 7-7 | [MultilingualReasoning](MultilingualReasoning/) | ✅ | 訓練模型在多語言環境下的推理能力，提升跨語言任務表現 |
| 7-9 | [cot-distillation](cot-distillation/) | ✅ | 經 OpenRouter 呼叫 Claude 等前沿模型蒸餾 CoT 軌跡，規則驗證器過濾後生成 SFT 資料（實驗 7-9 配套） |
| 7-10 | [AdaptThink](AdaptThink/) | 📖 | 讓推理模型按問題難度自適應選 Thinking/NoThinking，約束最佳化 + 重要性取樣降成本 45–69% 同時提升準確率 |
| 7-11 | `SFTvsRL/` | 📖 | 系統性對比監督微調與強化學習在不同任務上的效果與適用場景 |
| 7-12 | [SpatialReasoning](SpatialReasoning/) | 📖 | 訓練模型的空間推理能力，處理位置、方向、距離等空間關係 |
| 7-13 | [SimpleVLA-RL](SimpleVLA-RL/) | 📖 | 視覺-語言-動作 RL，讓模型理解視覺輸入並執行相應動作 |
| 7-14 | [retool](retool/) | 📖 | 多輪對話 + 程式碼沙箱提升數學推理，SFT→RL 兩階段；Qwen2.5-32B + AIME 2024 + DAPO + SandboxFusion |
| 7-15 | `AWorld/` · [AWorld-train](AWorld-train/) | 📖 | 基於 AWorld 框架訓練具身 Agent，在虛擬環境中執行任務並從經驗中學習 |
| 7-16 | [RLVP](RLVP/) | 📖 | 獎勵結果、懲罰路徑（RLVP）後訓練研究（實驗 7-16 配套）；完整訓練/評估程式碼在獨立論文倉庫 `19PINE-AI/rlvp`，需自行克隆 |
| 7-17 | [premature-completion-dpo](premature-completion-dpo/) | ✅ | 在 GPU 上以 DPO 修復過早完成 bad case |
| 7-18 | [curly-quote-sft](curly-quote-sft/) | ✅ | 經資料審核的作用域敏感中文彎引號 SFT：10 種文章體裁、9 種程式語言，train/holdout/boundary=1024/256/256；Qwen3-8B exact 96.9%/97.7%，保護區保留率 100% |
| 7-19 | [exact-copy-sft](exact-copy-sft/) | ✅ | 經資料審核的特殊字串 byte-exact 複製 SFT：1024/256/256 筆；Qwen3-8B holdout 78.9%、boundary 80.1%，另有 Qwen3/Qwen2.5/Mistral tokenizer 審核 |
| — | `verl/` | 📖 | 為 LLM RLHF 設計的高效 RL 框架，支援 PPO/GRPO/DAPO 等 |
| — | [Intuitor](Intuitor/) | ✅ | 訓練模型的直覺推理，快速做出合理判斷而不依賴詳細思考鏈 |
| — | `tinker-cookbook/` | 📖 | 收集各種模型訓練的實用技巧與最佳實踐 |
## 專案型別說明
| 圖示 | 型別 | 含義 |
| :--: | --- | --- |
| ✅ | **可獨立執行** | 本倉庫自帶完整程式碼，配置好 API Key 即可執行 |
| 📖 | **復現指南** | 依賴需自行 `git clone` 的**外部倉庫**（訓練框架、評測基準等） |
| 🚧 | **設計文件** | 僅包含架構與實現方案，可執行程式碼仍在完善中 |
chapter8/RLVP/README.md
+52
View File
@@ -0,0 +1,52 @@
# Experiment 8-16 reproduction anchor
The book-owned guide is this directory; executable code belongs in `chapter8/RLVP/rlvp` and comes from [`19PINE-AI/rlvp`](https://github.com/19PINE-AI/rlvp). The current workspace has no such checkout. A read-only upstream audit fixed revision `1ad30bc7e338911fb733739393d92c420f4d8bee` and verified the entrypoints below; no RLVP training or evaluation was run.
```bash
git clone https://github.com/19PINE-AI/rlvp.git chapter8/RLVP/rlvp
git -C chapter8/RLVP/rlvp fetch origin 1ad30bc7e338911fb733739393d92c420f4d8bee
git -C chapter8/RLVP/rlvp checkout --detach 1ad30bc7e338911fb733739393d92c420f4d8bee
git -C chapter8/RLVP/rlvp rev-parse HEAD
test "$(git -C chapter8/RLVP/rlvp rev-parse HEAD)" = "1ad30bc7e338911fb733739393d92c420f4d8bee"
```
At this revision, the audited sequence is `python3 tests/test_rules.py && python3 tests/test_credit.py`, `python3 scripts/phase0_baseline.py`, `bash scripts/run_all.sh`, and `python3 scripts/eval_checkpoint.py results/run_c3/final c3_norules`. The full campaign requires CUDA. The chapter's reported paper results are not a current-workspace execution claim.
## English
# Experiment 8-16: RLVP —— Reward the Outcome, Penalize the Path
> 📖 **The complete training/evaluation code corresponding to this experiment is in the standalone paper repository: [`github.com/19PINE-AI/rlvp`](https://github.com/19PINE-AI/rlvp)**
>
> RLVP (Reward the Outcome, Penalize the Path) is a post-training study by the author team. All results reported in Chapter 8, Experiment 8-16 of the book (violation rate, miniF2F, full-loss group proportion, etc.) come from experiments in that repository. Since training depends on GPU clusters and the code is continuously updated alongside the paper, the main book repository no longer duplicates it. Please go directly to the upstream repository for the latest code, configuration, and reproduction instructions:
```bash
# Use the pinned clone/fetch/detached-checkout/SHA-verification block above.
```
## Relationship with Other Training Experiments in This Chapter
This directory, like `chapter8/AdaptThink`, `chapter8/retool`, and `chapter8/AWorld-train`, belongs to the **reproduction guide (KEEP-EXT)**: the core training code resides in an external repository; simply follow its README to reproduce. For an explanation of the method and conclusions, see the corresponding section on "Model Post-Training" in Chapter 8 of the book.
---
## 中文
# 实验 8-16：RLVP —— 奖励结果、惩罚路径
> 📖 **本实验对应的完整训练/评估代码在独立论文仓库：[`github.com/19PINE-AI/rlvp`](https://github.com/19PINE-AI/rlvp)**
>
> RLVP（Reward the outcome, Penalize the path）是作者团队的一项后训练研究。书中第 7 章
> 实验 8-16 报告的各项结果（违规率、miniF2F、全败组占比等）均来自该仓库的实验。由于训练
> 依赖 GPU 集群、且代码随论文持续更新，本书主仓库不再重复内置，请直接前往上游仓库获取最新
> 代码、配置与复现说明：
```bash
# 请使用本 README 顶部固定版本的 clone/fetch/detached-checkout/SHA 校验命令。
```
## 与本章其它训练类实验的关系
本目录与 `chapter8/AdaptThink`、`chapter8/retool`、`chapter8/AWorld-train` 等一样，属于
**复现指南（KEEP-EXT）**：核心训练代码在外部仓库，按其 README 复现即可。书中对方法与结论的
讲解见正文第 7 章「模型后训练」相应小节。
chapter8/SimpleVLA-RL/README.md
+2040
View File
File diff suppressed because it is too large Load Diff
chapter8/SimpleVLA-RL/robotwin2_tasks_description.md
+702
View File
@@ -0,0 +1,702 @@
# RoboTwin 2.0 Tasks: Experimental Setup
**Tasks Used in Experiments**: `beat_block_hammer` and `move_can_pot`
**Based on**: [SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning](https://arxiv.org/pdf/2509.09674) (2025)
---
## Table of Contents
1. [Overview](#overview)
2. [Task 1: beat_block_hammer](#task-1-beat_block_hammer)
3. [Task 2: move_can_pot](#task-2-move_can_pot)
4. [Environment Randomization](#environment-randomization)
5. [Language Instruction System](#language-instruction-system)
6. [Training Configuration](#training-configuration)
---
## Overview
Both tasks are part of the RoboTwin 2.0 benchmark, designed for dual-arm manipulation with realistic physics simulation using SAPIEN. These tasks test different manipulation capabilities:
- **beat_block_hammer**: Tool use and contact-based manipulation
- **move_can_pot**: Pick-and-place with spatial reasoning
### Common Characteristics
| Property | Value |
|----------|-------|
| **Simulator** | SAPIEN (CPU-based physics) |
| **Robot Platform** | ALOHA (dual-arm, 7 DOF per arm) |
| **Action Space** | 14-dimensional continuous (7 DOF × 2 arms) |
| **Observation** | RGB images (3 cameras) + Proprioception (14-dim joint states) |
| **Camera Views** | Head camera + Left wrist + Right wrist (224×224×3 each) |
| **Max Environment Steps** | 200 steps per episode |
| **Action Chunking** | 25 action chunks per VLA inference |
| **VLA Calls per Episode** | ~8 (200 ÷ 25) |
| **Traj Mini-Batch Size** | 8 |
---
## Task 1: beat_block_hammer
### Task Description
**Objective**: Use the robot arm to grasp a hammer and strike a block on the table.
**Natural Language Description**: "There is a hammer and a block on the table, use the arm to grab the hammer and beat the block"
### Scene Setup
The environment contains two objects on a table:
#### Hammer
- **Model**: Standard hammer (model ID: 020_hammer)
- **Initial Position**: Fixed location at table center
- X-coordinate: 0 meters (table center)
- Y-coordinate: -0.06 meters (slightly back from center)
- Z-coordinate: 0.783 meters (on table surface)
- **Orientation**: Fixed (slightly angled)
- **Physical Properties**: Very light mass (0.001 kg) for easy manipulation
- **Functional Points**:
- Point 0: Hammer head (for striking)
- Base point: Handle (for grasping)
#### Block
- **Model**: Static red cube (doesn't move when hit)
- **Size**: 2.5 cm × 2.5 cm × 2.5 cm
- **Initial Position**: **Randomized** (see randomization section)
- **Color**: Red (for visibility)
- **Physical Properties**: Static (fixed to table, doesn't react to physics)
- **Functional Points**:
- Point 0: Block center
- Point 1: Block top surface (target for hammer)
### Success Criteria
The task is considered successful when **ALL** of the following conditions are met:
1. **Position Accuracy**: The hammer head is within 2 cm of the block position (in X-Y plane)
- Measured as Euclidean distance in horizontal plane
- Tolerance: ±2 cm in both X and Y directions
2. **Contact Detection**: The hammer and block are in physical contact
- Verified through SAPIEN's collision detection system
- Must be simultaneous with position accuracy
**Mathematical Formulation**:
```
Success = (|hammer_x - block_x| < 0.02 m) AND
(|hammer_y - block_y| < 0.02 m) AND
(hammer_contacts_block = True)
```
### Expert Demonstration Strategy
Human demonstrations follow a four-step procedure:
**Step 1: Arm Selection**
- Automatically select arm based on block position
- If block X-coordinate < 0: use left arm
- If block X-coordinate ≥ 0: use right arm
- This creates symmetric training data for both arms
**Step 2: Grasp Hammer**
- Approach hammer from 12 cm away (pre-grasp phase)
- Move gripper to handle position
- Close gripper at 1 cm distance (secure grasp)
- Gripper fully closes around handle
**Step 3: Lift Hammer**
- Raise hammer 7 cm upward (Z-axis displacement)
- Creates clearance from table surface
- Prepares for striking motion
**Step 4: Strike Block**
- Move hammer toward block's top surface
- Approach from 6 cm above (pre-placement phase)
- Lower hammer until contact with block
- Maintain closed gripper throughout strike
- Stop when contact is detected
### Challenges & Learning Opportunities
**Key Challenges**:
1. **Spatial Reasoning**: Block position varies significantly (50×20 cm range)
2. **Tool Use**: Must use hammer as extension of arm, not direct manipulation
3. **Contact Control**: Requires precise positioning for contact detection
4. **Arm Coordination**: Single-arm task but must avoid self-collision
**RL Discovery Potential**:
- SFT model learns: grasp → lift → move → strike (following demonstrations)
- RL may discover: more efficient striking trajectories, improved contact timing
- Demonstrates RL's ability to refine precision and timing in tool-use scenarios
---
## Task 2: move_can_pot
### Task Description
**Objective**: Pick up a can from the table and place it beside a pot.
**Natural Language Description**: "There is a can and a pot on the table, use one arm to pick up the can and move it to beside the pot"
### Scene Setup
The environment contains two objects on a table:
#### Pot (Kitchen Pot)
- **Model**: Kitchen pot with handles (model family: 060_kitchenpot)
- **Model Variants**: 7 different pot models (IDs 0-6)
- Different sizes, shapes, colors
- Randomly selected each episode
- **Initial Position**: Near table center
- X-coordinate: 0 meters (center)
- Y-coordinate: 0 meters (center)
- Z-coordinate: On table surface
- **Orientation**: Random rotation ±22.5° around vertical axis
- **Physical Properties**: Normal mass, can be moved if pushed hard
- **Functional Points**: Center point for distance measurement
#### Can (Sauce Can)
- **Model**: Cylindrical sauce can (model family: 105_sauce-can)
- **Model Variants**: 5 different can models (IDs: 0, 2, 4, 5, 6)
- Different labels, sizes
- Randomly selected from subset
- **Initial Position**: **Randomized** (see randomization section)
- **Orientation**: Random rotation around vertical axis
- **Physical Properties**: Standard mass, cylindrical shape
- **Functional Points**: Center point and top surface
### Success Criteria
The task is considered successful when **ALL** of the following conditions are met:
1. **Horizontal Distance**: Can is 18 cm (±20 cm tolerance) from pot horizontally
- Measured on the correct side (left arm → left of pot, right arm → right of pot)
- Must be on the correct side: cannot be on opposite side of pot
2. **Lateral Alignment**: Can Y-position is within 3.5 cm of pot Y-position
- Ensures can is "beside" pot, not in front or behind
3. **Can Orientation (Upright)**:
- X-axis rotation: 90° ± 15° (upright, not tilted forward/backward)
- Y-axis rotation: 0° ± 15° (upright, not tilted left/right)
- Can must maintain stable upright position
4. **On Table Surface**: Can Z-position ≤ original pot height + 0.1 cm
- Ensures can is resting on table, not floating or dropped
5. **Released**: Both robot grippers are open
- Confirms can has been released successfully
- Not being held by robot
**Mathematical Formulation**:
```
Success = (0 < |can_x - pot_x| < 0.2 m) AND # Distance tolerance
(|can_y - pot_y| < 0.035 m) AND # Lateral alignment
(|can_rotation_x - 90°| < 15°) AND # Upright X
(|can_rotation_y - 0°| < 15°) AND # Upright Y
(can_z ≤ table_surface + 0.001 m) AND # On surface
(can_on_correct_side = True) AND # Correct side of pot
(left_gripper_open = True) AND # Released
(right_gripper_open = True) # Released
```
### Expert Demonstration Strategy
Human demonstrations follow a five-step procedure:
**Step 1: Arm Selection**
- Automatically select arm based on can position
- If can X-coordinate > 0: use right arm
- If can X-coordinate ≤ 0: use left arm
- This determines which side of pot to place can
**Step 2: Grasp Can**
- Approach can from 5 cm away (pre-grasp phase)
- Move gripper to can center
- Close gripper around can body
- Secure cylindrical grasp
**Step 3: Lift and Retract**
- Move 10 cm backward (away from pot, Y-axis: -0.1 m)
- Simultaneously lift 10 cm upward (Z-axis: +0.1 m)
- Creates clearance from pot and other obstacles
- Prevents collision during transport
**Step 4: Transport to Target**
- Calculate target position beside pot:
- If left arm: target_x = pot_x - 0.18 m (18 cm to left)
- If right arm: target_x = pot_x + 0.18 m (18 cm to right)
- target_y = pot_y (same lateral position)
- target_z = table_surface
- Move gripper with can to target position
- Maintain upright orientation throughout
**Step 5: Place Can**
- Approach target from 5 cm above (pre-placement phase)
- Lower can smoothly to table surface
- Ensure stable contact with table
- Open gripper to release can
- Retract gripper away from can
### Challenges & Learning Opportunities
**Key Challenges**:
1. **Spatial Reasoning**: Understanding "beside" relationship (specific distance)
2. **Dual Randomization**: Both pot and can positions/models vary
3. **Orientation Maintenance**: Must keep can upright throughout manipulation
4. **Precision Placement**: Tight tolerance on final can orientation (±15°)
5. **Side Selection**: Must place on correct side based on arm used
**RL Discovery Potential - "Pushcut" Phenomenon Observed**:
- SFT learns: grasp → lift high → move → lower carefully (grasp-move-place strategy)
- **RL discovers: PUSH the can directly to target position** (pushcut strategy)
- This is a novel behavior NOT present in any demonstration data
- Pushcut strategy is faster and more robust than the demonstrated approach
- Paper Section 6.1: "the RL-trained model instead learns to accomplish the task by simply pushing Object A into position"
- Demonstrates emergence of completely new manipulation strategies through RL exploration
---
## Environment Randomization
### Purpose of Randomization
Randomization serves multiple purposes in robot learning:
1. **Prevent Overfitting**: Ensures policy learns general strategies, not memorizing specific positions
2. **Improve Generalization**: Policy must work across diverse configurations
3. **Test Robustness**: Validates policy can handle variability
4. **Enable Transfer**: Prepares policy for real-world deployment where exact positions cannot be controlled
### beat_block_hammer Randomization
#### Block Position (Fully Randomized)
The block position is randomly sampled each episode:
**X-Coordinate (Left-Right)**:
- Range: -0.25 to +0.25 meters (50 cm span)
- Distribution: Uniform random
- Constraints: Must not be too close to center (|x| ≥ 0.05 m)
- Purpose: Tests both left and right arm capabilities
**Y-Coordinate (Front-Back)**:
- Range: -0.05 to +0.15 meters (20 cm span)
- Distribution: Uniform random
- Constraints: None specific
- Purpose: Tests depth perception and reaching
**Z-Coordinate (Height)**:
- Fixed: 0.76 meters (on table surface)
- No variation: Block always at same height
**Orientation (Rotation)**:
- Rotation around Z-axis: 0 to 0.5 radians (0° to ~29°)
- Distribution: Uniform random
- Purpose: Tests rotational invariance
- Note: Block is square, so rotation matters less
**Additional Constraints**:
- Block must be at least 5 cm from table center (|x| ≥ 0.05)
- Block must be at least 0.1 cm from origin (x² + y² ≥ 0.001)
- These prevent trivial configurations
**Prohibited Areas**:
- 10 cm padding around hammer position
- 5 cm padding around block position
- Robot must plan path avoiding these zones
#### Hammer Position (Fixed)
The hammer position is **not randomized**:
- Always at same position: [0, -0.06, 0.783] meters
- Fixed orientation
- Rationale: Simplifies task to focus on block positioning challenge
**Randomization Summary for beat_block_hammer**:
```
Total Randomized Parameters: 3
- Block X: 50 cm range
- Block Y: 20 cm range
- Block Rotation: ~29° range
Configuration Space: ~50 × 20 × 29 = 29,000 possible discrete configurations
Continuous Space: Infinite (continuous sampling)
```
### move_can_pot Randomization
#### Pot Randomization (Partial)
**Model Variant**:
- Randomly select from 7 pot models (IDs 0-6)
- Different visual appearances (colors, sizes, shapes)
- Tests visual generalization
**Position**:
- X-coordinate: 0 meters (fixed at center)
- Y-coordinate: 0 meters (fixed at center)
- Z-coordinate: On table surface (fixed)
**Orientation**:
- Rotation around Z-axis: ±22.5° (±π/8 radians)
- Distribution: Uniform random
- Purpose: Tests rotation invariance for target reference
#### Can Randomization (Full)
**Model Variant**:
- Randomly select from 5 can models (IDs: 0, 2, 4, 5, 6)
- Different labels and appearances
- Note: IDs 1 and 3 are excluded (possibly unstable or problematic)
**Position**:
- X-coordinate: -0.3 to +0.3 meters (60 cm span)
- Constraint: |x| ≥ 0.2 m (must not be too close to center)
- Prevents can from being directly in front of robot
- Y-coordinate: +0.05 to +0.15 meters (10 cm span, front of table)
- Always on front side of table
- Easier for robot to reach
- Z-coordinate: On table surface (computed based on can model)
**Orientation**:
- Rotation around Z-axis: ±45° (±π/4 radians)
- Distribution: Uniform random
- Purpose: Tests grasping from different angles
**Spatial Constraints**:
- Can must be at least 30 cm from pot center
- Formula: (can_x - pot_x)² + (can_y - pot_y)² ≥ 0.09 m²
- Prevents can from starting too close to target
- If constraint violated, resample position
**Prohibited Areas**:
- 3 cm padding around pot position
- 10 cm padding around can position
- 15 cm × 20 cm zone on approach side of pot (arm-dependent)
- If left arm: zone extends 15 cm to left of pot
- If right arm: zone extends 15 cm to right of pot
- Prevents trivial straight-line motions
**Randomization Summary for move_can_pot**:
```
Total Randomized Parameters: 7
- Pot Model: 7 variants
- Pot Rotation: ±22.5°
- Can Model: 5 variants
- Can X: 60 cm range (with constraints)
- Can Y: 10 cm range
- Can Rotation: ±45°
Configuration Space: 7 × 45° × 5 × 60 × 10 × 90° ≈ 8.5M possible discrete configurations
Continuous Space: Infinite (continuous sampling)
```
### Randomization Process
Each episode follows this randomization procedure:
**Episode Start**:
1. Sample all random parameters from their respective distributions
2. Check spatial constraints (distance requirements, prohibited zones)
3. If constraints violated: resample violating parameters
4. Repeat until valid configuration found
5. Initialize SAPIEN scene with sampled configuration
6. Generate task instruction (random selection from template bank)
7. Compute initial observation (RGB images + proprioception)
8. Begin episode execution
**Key Design Principles**:
- **Constraint Validation**: Ensures physically feasible configurations
- **Iterative Sampling**: Rejection sampling until valid
- **Symmetric Design**: Equal probability for left/right arm usage
- **Difficulty Calibration**: Constraints prevent too-easy or impossible scenarios
### Impact on RL Training
**Dynamic Sampling Interaction**:
- Randomization creates natural difficulty distribution
- Dynamic Sampling (paper Section 3.3) further filters:
- Excludes episodes where all 8 samples succeed (too easy)
- Excludes episodes where all 8 samples fail (too hard)
- Keeps only episodes with mixed outcomes (learnable)
- Combined effect: RL focuses on appropriate difficulty frontier
**Curriculum Learning Effect**:
- Early training: High failure rate, mostly hard configurations kept
- Mid training: More mixed outcomes, broader configuration range
- Late training: High success rate, need new challenging configurations
- This naturally implements curriculum without manual intervention
---
## Language Instruction System
### Instruction Generation Pipeline
Both tasks use a sophisticated language instruction system to test generalization:
**Template-Based Generation**:
1. Define full task description (detailed)
2. Specify instruction preferences (word count, style)
3. Define object schema (placeholders for specific objects)
4. Use LLM to generate diverse paraphrases
5. Manually curate into "seen" and "unseen" sets
**Schema Placeholders**:
- `{A}`: Primary object (hammer in task 1, pot in task 2)
- `{B}`: Secondary object (can in task 2)
- `{a}`: Arm designation (left or right)
### beat_block_hammer Instructions
**Full Description**: "there is a hammer and a block on the table, use the arm to grab the hammer and beat the block"
**Schema**:
- `{A}` → hammer identifier (e.g., "020_hammer/base0")
- `{a}` → arm used (e.g., "left" or "right")
**Instruction Preferences**:
- Word count: ≤10 words
- Must mention: grasping action, striking action
- Can mention: arm designation (optional)
**Seen Instructions (50+ variants)**:
Examples used during training:
- "Pick {A} and strike the block."
- "Lift {A} using {a} to hit the block."
- "Grab {A} with {a} and beat the block."
- "Use {A} to hammer the block."
- "Take {A} and smash the block."
- "Hold {A} and pound the block."
**Unseen Instructions (10 variants)**:
Examples reserved for evaluation:
- "Grab {A} and beat the block."
- "Use {a} to pick up {A}."
- "Grab {A} and strike the block."
- "Use {a} to grab {A} then beat block"
**Instruction Diversity**:
- Verbs: grab, pick, lift, take, hold, use, employ
- Actions: beat, strike, hit, hammer, pound, smash
- Styles: imperative, sequential, compound sentences
- Tests: synonym understanding, sentence structure variations
### move_can_pot Instructions
**Full Description**: "there is a can and a pot on the table, use one arm to pick up the can and move it to beside the pot"
**Schema**:
- `{A}` → pot identifier (e.g., "060_kitchenpot/base3")
- `{B}` → can identifier (e.g., "105_sauce-can/base2")
- `{a}` → arm used (e.g., "left" or "right")
**Instruction Preferences**:
- Word count: ≤10 words
- Must mention: picking action, placement location
- Must specify: relative position to pot
**Seen Instructions (50+ variants)**:
Examples used during training:
- "Use {a} to grab {B} and move it next to {A}"
- "Pick {B} up with {a} then place near {A}"
- "Grab {B} with {a} and set it near {A}"
- "Lift {B} and move it near {A}"
- "Take {B} to {A} using {a}"
- "Set {B} right next to {A}"
**Unseen Instructions (10 variants)**:
Examples reserved for evaluation:
- "Pick up {B} and move it near {A}"
- "Grab {B} and set it beside {A}"
- "Use {a} to pick up {B}, move it near {A}"
- "Grab {B} and place it beside {A}"
**Instruction Diversity**:
- Verbs: grab, pick, lift, take, move, transfer, relocate
- Spatial terms: near, beside, next to, by, close to
- Styles: with/without arm mention, sequential vs compound
- Tests: spatial relationship understanding, action sequence parsing
### Episode-Level Instruction Assignment
**During Training** (Seen Instructions):
- Each rollout: randomly select one instruction from seen set
- 50+ instructions → low probability of repeating same instruction
- Each instruction has equal selection probability
- Ensures model doesn't overfit to specific phrasings
**During Validation** (Mixed):
- IID validation: use seen instructions (same distribution as training)
- OOD validation: use unseen instructions (test generalization)
- Both sets used to measure different aspects of performance
**Runtime Substitution**:
When instruction is selected, placeholders are replaced:
- `{A}` → actual object identifier from episode
- `{B}` → actual object identifier from episode
- `{a}` → "left" or "right" based on arm selection algorithm
**Example Runtime Process**:
```
Template: "Use {a} to grab {B} and move it next to {A}"
Episode Configuration:
- Pot model: 060_kitchenpot/base3
- Can model: 105_sauce-can/base5
- Can X-position: +0.22 (right side)
- Selected arm: right
Final Instruction: "Use right to grab 105_sauce-can/base5 and move it next to 060_kitchenpot/base3"
Simplified for VLA: "Use right arm to grab the sauce can and move it next to the kitchen pot"
```
### Instruction System Benefits
**For Training**:
1. **Language Generalization**: 50+ variants prevent language overfitting
2. **Robustness**: Model must understand task concept, not memorize phrases
3. **Diversity**: Different instruction styles cover various language patterns
4. **Grounding**: Schema system links language to specific objects in scene
**For Evaluation**:
1. **Unseen Instructions**: Tests true language understanding
2. **IID vs OOD**: Measures both in-distribution and out-of-distribution performance
3. **Zero-Shot Transfer**: Unseen instructions never appeared in training
4. **Generalization Metric**: Success rate on unseen instructions indicates robustness
---
## Training Configuration
### Dataset Configuration
**Training Data Source**:
- Pre-collected feasible seeds: 1000 per task
- Seeds validated through simulation to ensure solvability
- Stored in: `verl/utils/envs/robotwin2/seeds/robotwin2_train_seeds.json`
**Validation Data**:
- IID Validation: 128 seeds from training distribution
- OOD Validation: 128 seeds from held-out distribution
- Total validation: 256 episodes per task
**Data Loading**:
- Batch size: 64 task instances per training step
- Samples per task: 8 (for GRPO grouping)
- Total rollouts per step: 512 (64 × 8)
- Shuffle: Enabled (random task ordering each epoch)
### Rollout Configuration
**Action Space**:
- Continuous 14-dimensional actions
- 7 DOF per arm (position + orientation + gripper)
- Action normalization: [-1, 1] range
- Denormalization statistics: Task-specific (learned from demonstrations)
**Action Chunking**:
- Chunks per VLA inference: 25
- Action tokens per chunk: 14 (one per dimension)
- Total tokens per inference: 350 (25 × 14)
- VLA calls per episode: ~8 (200 steps ÷ 25)
**Rollout Parameters**:
- Temperature: 1.6 (Higher Rollout Temperature - paper enhancement)
- Sampling: Enabled (do_sample=True)
- Max environment steps: 200
- Micro-batch size: 1 (sequential processing)
- Episode timeout: 200 steps = success or failure
### Success Rate Targets
Based on paper results and dynamic sampling:
**beat_block_hammer**:
- Initial SFT performance: ~25-35% (estimated)
- Target after 300 steps: ~50-70% (paper-level)
- Dynamic sampling keeps: 10-90% success rate episodes
- Expected at convergence: ~60-80% success
**move_can_pot**:
- Initial SFT performance: ~20-30% (estimated)
- Target after 300 steps: ~45-65% (paper-level)
- Dynamic sampling keeps: 10-90% success rate episodes
- Expected at convergence: ~55-75% success
**Training Progress Indicators**:
- Epoch 0-5: Mostly failures, learning basic grasping
- Epoch 5-10: Increasing success, learning task structure
- Epoch 10-15: Rapid improvement, discovering strategies
- Epoch 15-20: Convergence, refining precision
- Beyond 20: Marginal gains, potential overfitting
### Computational Requirements
**Per Training Step** (~20 minutes):
- Rollout phase: ~18 minutes (87.8% of time)
- Environment initialization: ~2 minutes
- VLA inference: ~2 minutes (8 calls × 300ms × 512 rollouts / 8 GPUs)
- Physics simulation: ~14 minutes (25 steps × 60ms × 512 rollouts / 8 GPUs)
- PPO update phase: ~2.4 minutes (11.8% of time)
- Overhead: ~0.2 minutes (0.4% of time)
**Full Training** (300 steps as per paper):
- Total time: ~300 steps × 20 minutes = 6000 minutes ≈ 100 hours ≈ 4.3 days
- GPU usage: 8 × A100 GPUs (or equivalent)
- Total rollouts: 300 × 512 = 153,600 episodes
- Total VLA inferences: ~153,600 × 8 = ~1.2M forward passes
- Total environment steps: ~153,600 × 100 = ~15M physics steps (average)
**Resource Distribution**:
- GPU compute: ~20% utilization (during VLA inference only)
- CPU compute: ~80% utilization (physics simulation dominates)
- GPU memory: ~6 GB per GPU (FSDP sharding)
- System memory: ~64 GB (environment states, buffers)
---
## Summary
### Task Comparison
| Aspect | beat_block_hammer | move_can_pot |
|--------|------------------|--------------|
| **Skill Type** | Tool use + Contact | Pick & Place + Spatial |
| **Complexity** | Medium | Medium-High |
| **Randomization** | 1 object (block) | 2 objects (pot + can) |
| **Success Tolerance** | Tight (±2 cm) | Moderate (±20 cm horizontal) |
| **Key Challenge** | Precise contact | Spatial reasoning |
| **Expected Success** | 60-80% after RL | 55-75% after RL |
| **Novel Behaviors** | "Pushcut" (push vs lift) | Efficient low trajectories |
### Key Experimental Insights
**Why These Tasks?**
1. **Diverse Skills**: Test different manipulation capabilities (tool use vs spatial reasoning)
2. **Realistic Complexity**: Neither too easy nor impossible, appropriate for RL learning
3. **Measurable Success**: Clear binary success criteria, no ambiguity
4. **Rich Randomization**: Sufficient variability to test generalization
5. **Dual-Arm Platform**: Tests coordination and arm selection logic
**What RL Learns**:
1. **Beyond Demonstrations**: Discovers strategies not in expert data ("pushcut")
2. **Robustness**: Handles high variability through 153k diverse rollouts
3. **Efficiency**: Finds shorter, more direct paths than demonstrations
4. **Generalization**: Works on unseen instructions and configurations
5. **Exploration**: Higher temperature (1.6) enables diverse strategy discovery
**Expected Outcomes**:
- Both tasks show improvement with RL over SFT baseline
- "Pushcut" phenomenon may emerge in beat_block_hammer
- move_can_pot benefits from optimized trajectories
- Success rates reach 60-75% range after 300 training steps
- Unseen instructions show strong generalization (within 5-10% of seen)
chapter8/SimpleVLA-RL/verl_training_process.md
+1520
View File
File diff suppressed because it is too large Load Diff
chapter8/SimpleVLA-RL/vla-rollout-analysis.md
+1283
View File
File diff suppressed because it is too large Load Diff
chapter8/SpatialReasoning/README.md
+4990
View File
File diff suppressed because it is too large Load Diff
chapter8/continued-pretraining/README.md
+347
View File
@@ -0,0 +1,347 @@
## English
# Continued Pretraining: Teaching a Model a New Language (Korean Mistral)
> This directory corresponds to Chapter 7, **Experiment 8-5 ★★: Continued Pretraining for Learning a New Language** of *Deep Understanding of AI Agents*.
## Project Overview
Using **Mistral 7B v0.3** as the base model (primarily pretrained on English, with virtually no understanding of Korean), we inject Korean language capability through **continued pretraining on Korean Wikipedia**, followed by **SFT on Korean instruction data**. The final model can both understand Korean and follow instructions in Korean.
The core idea this experiment aims to demonstrate: **To make a model memorize a large amount of new domain knowledge (here, a new language), rely on continued pretraining, not SFT.** The model already possesses general language modeling ability from the pretraining phase; continued pretraining merely adapts it to a new data distribution, at a cost far lower than training from scratch.
The entire process consists of two stages:
1. **Continued Pretraining**: Unsupervised "predict the next token" training on Korean Wikipedia, allowing the model to learn Korean vocabulary and syntax.
2. **Instruction Fine-Tuning (SFT)**: Training on Korean Alpaca instruction data to teach the model to "follow instructions in Korean."
A key engineering challenge is **mitigating Catastrophic Forgetting**: learning a new language should not cause the model to forget its original English ability. The common approach discussed in the book uses mixed data (approximately 80% target language + 20% original language) to balance this; this implementation adopts a parameter-efficient scheme using **LoRA + training `embed_tokens`/`lm_head`** — only updating the adapters and word embeddings while keeping the base weights unchanged, thereby preserving English as much as possible while injecting Korean. Evaluation results (see below) show that English ability is largely retained.
## Directory Structure
```
continued-pretraining/
├── README.md # This document
├── continued-pretrain.py # Main training script: continued pretraining + SFT, produces two LoRA models
├── evaluate_model.py # Single model evaluation: generates samples on Korean/English tasks
├── compare_models.py # Three-stage comparison: base → continued pretraining → instruction fine-tuning side-by-side generation
├── model_eval_results.md # Full evaluation output and conclusions from actual run (RTX 4090)
├── validation/ # Canonical report audit, blind-judge receipts, manifest, and validator
├── README_EVALUATION.md # Detailed usage instructions for evaluation scripts
└── requirements.txt # Dependency list
```
Running the training script produces two local directories (saving only LoRA adapters, not the full model):
- `lora_model_pretrained/`: Model after continued pretraining, before SFT
- `lora_model/`: Model after final instruction fine-tuning
## System Requirements & Dependencies
- **GPU**: Requires a CUDA-capable NVIDIA GPU. By default, Mistral-7B is loaded in 4-bit quantization, allowing training on consumer-grade GPUs with approximately 24GB VRAM (e.g., RTX 4090). The results in `model_eval_results.md` were produced on an RTX 4090.
- **Framework**: [Unsloth](https://github.com/unslothai/unsloth) (efficient LoRA training), PyTorch, Transformers, Datasets, bitsandbytes.
- **Optional**: wandb (experiment tracking; script defaults to `report_to="wandb"`).
```bash
# From the repository root: use the shared Chapter 7 environment plus Unsloth
uv sync --locked --python 3.12 --extra ch7 --extra unsloth
# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat
# pip fallback when uv is not installed:
# python -m pip install -e ".[ch7,unsloth]"
cd chapter8/continued-pretraining
# Single-project compatibility path, still supported for exact legacy parity
# (including the original Unsloth Git install used by this project):
# python -m pip install -r requirements.txt
```
> Note: Unsloth depends on a GPU with a compatible CUDA/PyTorch version and cannot be used for training or inference in a pure CPU environment. The `--help` for each script uses lazy imports, so parameter descriptions can be viewed on machines without a GPU.
## Quick Start
### 1. Training (Continued Pretraining + SFT)
Run both stages with default hyperparameters in one command (using a 5% subset of Korean Wikipedia for continued pretraining, followed by SFT on Korean Alpaca):
```bash
python continued-pretrain.py
```
The script will sequentially: load the base model → print a baseline test → perform continued pretraining on Korean Wikipedia → save `lora_model_pretrained/` → perform SFT on Korean instructions → save `lora_model/`.
Common parameters (defaults match the script's original hardcoded values; changes will deviate from the original experiment):
```bash
python continued-pretrain.py \
--base_model unsloth/mistral-7b-v0.3 \
--wiki_config 20231101.ko \
--wiki_train_size 0.05 \
--alpaca_dataset FreedomIntelligence/alpaca-gpt4-korean \
--lora_rank 128 \
--max_seq_len 2048 \
--pretrain_epochs 1 \
--sft_epochs 2 \
--pretrained_save_dir lora_model_pretrained \
--final_save_dir lora_model
```
- For a quick smoke test, use `--pretrain_max_steps 20 --sft_max_steps 20` to run only a few steps.
- To switch to a different language: replace `--wiki_config` with the corresponding Wikipedia snapshot (e.g., `20231101.ja` for Japanese) and `--alpaca_dataset` with the corresponding instruction dataset.
- See `python continued-pretrain.py --help` for the full list of parameters.
### 2. Evaluating a Single Model
```bash
# Evaluate the final fine-tuned model (default loads lora_model/)
python evaluate_model.py
# Evaluate the model after continued pretraining, before SFT
python evaluate_model.py --pretrained
# Generate longer outputs using sampling
python evaluate_model.py --max_new_tokens 300 --use_sampling --temperature 0.7
```
See [`README_EVALUATION.md`](./README_EVALUATION.md) for more usage details.
### 3. Three-Stage Side-by-Side Comparison
Load the **base model / continued pretraining model / instruction fine-tuning model** simultaneously and generate side-by-side outputs on the same set of Korean and English prompts, visually demonstrating the improvement in Korean ability and the retention of English ability:
```bash
python compare_models.py
```
```bash
# Specify model directories and generation parameters
python compare_models.py \
--pretrained_path lora_model_pretrained \
--finetuned_path lora_model \
--max_new_tokens 150 \
--temperature 0.3
```
## Experimental Results
The full terminal output from the historical RTX 4090 run is retained in [`model_eval_results.md`](./model_eval_results.md). The canonical evidence package is [`validation/runs/exp8-5-training-report-20260731-v1/`](validation/runs/exp8-5-training-report-20260731-v1/), and [`validation/latest.json`](validation/latest.json) binds its manifest.
The audit extracted all **5 prompts × 3 stages = 15 outputs** and sent five deterministic stage-blind comparison tasks to the independent ARK `doubao-seed-1-6-250615` judge. All five raw request/responses, unique response IDs, usage, and latency are retained. The 0–5 mean scores were:
| Stage | Korean | English |
| --- | ---: | ---: |
| Base Mistral | 1.6667 | 5.0000 |
| After Korean continued pretraining | 1.3333 | 3.1667 |
| After Korean instruction SFT | 3.4444 | 4.1667 |
The final stage improved the Korean mean by **+1.7777** over baseline. Its English mean fell by **0.8333**, within the audit's declared 1.0-point retention tolerance. Continued pretraining alone did not improve this small retained prompt set; the final SFT stage produced the observed Korean gain. The kimchi answer remained materially false: it described boiling vegetables and soaking them in a soy-sauce-based sauce. That limitation is an accepted negative result, not hidden by the aggregate score.
The historical run did not retain adapter hashes, exact resolved upstream commits, or its generation seed. For future reproduction, the audit freezes immutable current revisions for the base model and both datasets in [`reproduction_contract.json`](validation/runs/exp8-5-training-report-20260731-v1/reproduction_contract.json). Those pins are explicitly **not claimed to be the historical revisions**.
Training checkpoints/adapters are intentionally local and are not distributed with the book. They are not acceptance artifacts; the accepted artifact is the reproducible, evidence-backed report. Validate it without a GPU or provider call:
```bash
python chapter8/continued-pretraining/validation/validate_evidence.py
python -m pytest chapter8/continued-pretraining/validation/test_report_audit.py -q
```
To create a new independent audit from the retained report, set `ARK_API_KEY` and use a new run ID:
```bash
python chapter8/continued-pretraining/validation/run_report_audit.py \
--run-id exp8-5-training-report-YYYYMMDD-vN
```
The evidence-backed conclusions are:
- **The full two-stage path improved Korean in this retained comparison**: the final SFT stage scored substantially above the baseline, while the continued-pretrained intermediate stage did not.
- **English remained usable but measurably regressed**: the final stage stayed within the declared tolerance; the intermediate stage regressed much more.
- **Fluency is not factual reliability**: the fluent final kimchi answer contains serious preparation and ingredient errors.
## References
- Unsloth documentation: https://docs.unsloth.ai
- Base model: [unsloth/mistral-7b-v0.3](https://huggingface.co/unsloth/mistral-7b-v0.3)
- Continued pretraining corpus: [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (`20231101.ko`)
- Instruction fine-tuning corpus: [FreedomIntelligence/alpaca-gpt4-korean](https://huggingface.co/datasets/FreedomIntelligence/alpaca-gpt4-korean)
---
## 中文
# 继续预训练：让模型学会一门新语言（韩语 Mistral）
> 本目录对应《深入理解 AI Agent》第 7 章 **实验 8-5 ★★：继续预训练学习新语言**。
## 项目简介
以 **Mistral 7B v0.3** 为基础模型（主要用英语预训练，对韩语几乎没有理解能力），通过**韩语维基百科继续预训练**注入韩语能力，再用**韩语指令数据做 SFT**，最终得到一个既能理解韩语、又能用韩语遵循指令的模型。
本实验想说明的核心观点：**要让模型记住大量新领域知识（这里是一门新语言），靠的是继续预训练，而不是 SFT。** 模型在预训练阶段已经具备通用的语言建模能力，继续预训练只是让它适应新的数据分布，成本远低于从头训练。
整个流程分两个阶段：
1. **继续预训练（Continued Pretraining）**：在韩语维基百科上做无监督的“预测下一个词”训练，让模型学会韩语的词汇与句法。
2. **指令微调（SFT）**：在韩语 Alpaca 指令数据上训练，让模型学会“用韩语遵循指令”。
一个关键工程点是**缓解灾难性遗忘（Catastrophic Forgetting）**：学了新语言不能把原来的英语能力忘掉。书中讨论的通用做法是用混合数据（约 80% 目标语言 + 20% 原语言）来平衡；本实现则采用 **LoRA + 训练 `embed_tokens`/`lm_head`** 的参数高效方案——只更新适配器与词嵌入，基础权重保持不变，从而在注入韩语的同时尽量保留英语。评测结果（见下文）显示英语能力基本得到保留。
## 目录结构
```
continued-pretraining/
├── README.md # 本文档
├── continued-pretrain.py # 训练主脚本：继续预训练 + SFT，产出两个 LoRA 模型
├── evaluate_model.py # 单模型评测：在韩英任务上生成样例
├── compare_models.py # 三阶段对比：基础 → 继续预训练 → 指令微调 并排生成
├── model_eval_results.md # 真实运行的完整评测输出与结论（RTX 4090）
├── validation/ # 规范报告审计、盲评回执、manifest 与验证器
├── README_EVALUATION.md # 评测脚本的详细用法说明
└── requirements.txt # 依赖清单
```
训练脚本运行后会产出两个本地目录（仅保存 LoRA 适配器，不含完整模型）：
- `lora_model_pretrained/`：继续预训练之后、SFT 之前的模型
- `lora_model/`：最终指令微调之后的模型
## 系统要求与依赖
- **GPU**：需要支持 CUDA 的 NVIDIA GPU。默认以 4bit 量化加载 Mistral-7B，可在约 24GB 显存的消费级显卡（如 RTX 4090）上完成训练，`model_eval_results.md` 中的结果即在 RTX 4090 上产出。
- **框架**：[Unsloth](https://github.com/unslothai/unsloth)（高效 LoRA 训练）、PyTorch、Transformers、Datasets、bitsandbytes。
- **可选**：wandb（实验跟踪，脚本默认 `report_to="wandb"`）。
```bash
# 在仓库根目录使用统一的第 7 章环境，并显式加入 Unsloth
uv sync --locked --python 3.12 --extra ch7 --extra unsloth
# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat
# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch7,unsloth]"
cd chapter8/continued-pretraining
# 迁移期间仍支持单项目兼容路径，用于完全复现旧版依赖
#（包括本项目原有的 Unsloth Git 安装方式）：
# python -m pip install -r requirements.txt
```
> 注意：Unsloth 依赖 GPU 与匹配的 CUDA/PyTorch 版本，无法在纯 CPU 环境下训练或推理。各脚本的 `--help` 已做延迟导入，可在没有 GPU 的机器上直接查看参数说明。
## 快速开始
### 1. 训练（继续预训练 + SFT）
用默认超参数一键完成两个阶段（韩语维基百科 5% 子集做继续预训练，随后用韩语 Alpaca 做 SFT）：
```bash
python continued-pretrain.py
```
脚本会依次：加载基础模型 → 打印基线测试 → 韩语维基继续预训练 → 保存 `lora_model_pretrained/` → 韩语指令 SFT → 保存 `lora_model/`。
常用参数（默认值与脚本原始硬编码一致，改动才会偏离原实验）：
```bash
python continued-pretrain.py \
--base_model unsloth/mistral-7b-v0.3 \
--wiki_config 20231101.ko \
--wiki_train_size 0.05 \
--alpaca_dataset FreedomIntelligence/alpaca-gpt4-korean \
--lora_rank 128 \
--max_seq_len 2048 \
--pretrain_epochs 1 \
--sft_epochs 2 \
--pretrained_save_dir lora_model_pretrained \
--final_save_dir lora_model
```
- 想快速冒烟测试，可用 `--pretrain_max_steps 20 --sft_max_steps 20` 只跑很少的步数。
- 想换一门语言：把 `--wiki_config` 换成对应维基快照（如 `20231101.ja` 日语）、`--alpaca_dataset` 换成对应语言的指令集即可。
- 完整参数见 `python continued-pretrain.py --help`。
### 2. 评测单个模型
```bash
# 评测最终微调模型（默认加载 lora_model/）
python evaluate_model.py
# 评测继续预训练后、SFT 前的模型
python evaluate_model.py --pretrained
# 生成更长、使用采样
python evaluate_model.py --max_new_tokens 300 --use_sampling --temperature 0.7
```
更多用法详见 [`README_EVALUATION.md`](./README_EVALUATION.md)。
### 3. 三阶段并排对比
同时加载**基础模型 / 继续预训练模型 / 指令微调模型**，在同一组中韩英提示上并排生成，直观展示韩语能力的提升与英语能力的保留：
```bash
python compare_models.py
```
```bash
# 指定模型目录与生成参数
python compare_models.py \
--pretrained_path lora_model_pretrained \
--finetuned_path lora_model \
--max_new_tokens 150 \
--temperature 0.3
```
## 实验结果
历史 RTX 4090 运行的完整终端输出保存在 [`model_eval_results.md`](./model_eval_results.md)。规范证据包位于 [`validation/runs/exp8-5-training-report-20260731-v1/`](validation/runs/exp8-5-training-report-20260731-v1/)，[`validation/latest.json`](validation/latest.json) 绑定其 manifest。
审计从原始报告提取了 **5 个提示 × 3 个阶段 = 15 个输出**，并向独立 ARK `doubao-seed-1-6-250615` 裁判提交了五次确定性乱序、阶段匿名的对比。五份原始请求/响应、唯一 response ID、usage 与延迟均已保留。0–5 分均值如下：
| 阶段 | 韩语 | 英语 |
| --- | ---: | ---: |
| 基础 Mistral | 1.6667 | 5.0000 |
| 韩语继续预训练后 | 1.3333 | 3.1667 |
| 韩语指令 SFT 后 | 3.4444 | 4.1667 |
最终阶段相对基线的韩语均值提升 **+1.7777**；英语均值下降 **0.8333**，仍在预先声明的 1.0 分保留容差内。仅继续预训练的中间阶段在这组小规模保留提示上没有提升，观察到的韩语增益来自完整两阶段流程后的最终 SFT 模型。最终模型的泡菜回答仍有严重事实错误：它错误地描述了煮蔬菜和以酱油为基础的浸泡汁。这个负结果被明确保留，而没有被总分掩盖。
历史运行没有保留 adapter hash、当时解析到的上游 commit 或生成随机种子。为了将来复现，[`reproduction_contract.json`](validation/runs/exp8-5-training-report-20260731-v1/reproduction_contract.json) 固定了基础模型和两个数据集的当前不可变 revision；这些 revision 明确**不声称是历史运行所用版本**。
训练 checkpoint/adapter 按本书策略仅保存在本地，不随书分发，也不是验收产物；验收产物是可复现、证据充分的训练报告。无需 GPU 或 API 调用即可验证：
```bash
python chapter8/continued-pretraining/validation/validate_evidence.py
python -m pytest chapter8/continued-pretraining/validation/test_report_audit.py -q
```
如需从保留报告创建新的独立审计，设置 `ARK_API_KEY` 并使用新的 run ID：
```bash
python chapter8/continued-pretraining/validation/run_report_audit.py \
--run-id exp8-5-training-report-YYYYMMDD-vN
```
有证据支持的结论如下：
- **完整两阶段流程在本次保留对比中提升了韩语**：最终 SFT 阶段显著高于基线，但继续预训练的中间阶段没有提升。
- **英语仍可用，但出现可测量的退化**：最终阶段仍在声明容差内；中间阶段退化更明显。
- **流畅不等于事实可靠**：最终泡菜回答虽然更流畅，却包含严重的制作方法与配料错误。
## 参考资料
- Unsloth 文档：https://docs.unsloth.ai
- 基础模型：[unsloth/mistral-7b-v0.3](https://huggingface.co/unsloth/mistral-7b-v0.3)
- 继续预训练语料：[wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)（`20231101.ko`）
- 指令微调语料：[FreedomIntelligence/alpaca-gpt4-korean](https://huggingface.co/datasets/FreedomIntelligence/alpaca-gpt4-korean)
chapter8/continued-pretraining/README_EVALUATION.md
+247
View File
@@ -0,0 +1,247 @@
# Korean Mistral Model Evaluation Guide
This guide explains how to use the evaluation script to test your trained Korean Mistral models.
## Overview
After running `continued-pretrain.py`, you'll have two saved models:
- `lora_model_pretrained/` - Model after Korean pretraining (before instruction finetuning)
- `lora_model/` - Final model after instruction finetuning
## Quick Start
### Basic Evaluation (Final Finetuned Model)
```bash
python evaluate_model.py
```
This will:
- Load the final finetuned model from `lora_model/`
- Run 6 test cases (Korean + English, Wikipedia + Instructions)
- Use default parameters (max_new_tokens=150)
### Evaluate Pretrained Model (Before SFT)
```bash
python evaluate_model.py --pretrained
```
This loads the model after Korean pretraining but before instruction finetuning.
## Command Line Options
### Model Selection
```bash
# Evaluate the pretrained model
python evaluate_model.py --pretrained
# Evaluate a custom model path
python evaluate_model.py --model_path path/to/your/model
# Load in full precision (more memory, higher quality)
python evaluate_model.py --load_in_4bit False
```
### Generation Parameters
```bash
# Generate more tokens
python evaluate_model.py --max_new_tokens 300
# Use sampling for more creative outputs
python evaluate_model.py --use_sampling --temperature 0.8 --top_p 0.95
```
### All Available Options
| Option | Default | Description |
|--------|---------|-------------|
| `--model_path` | `lora_model` | Path to saved LoRA model |
| `--pretrained` | `False` | Load pretrained model (before SFT) |
| `--max_seq_length` | `2048` | Maximum sequence length |
| `--load_in_4bit` | `True` | Use 4-bit quantization |
| `--max_new_tokens` | `150` | Maximum tokens to generate |
| `--use_sampling` | `False` | Enable sampling (vs greedy) |
| `--temperature` | `0.7` | Sampling temperature (creativity) |
| `--top_p` | `0.9` | Top-p nucleus sampling |
## Example Use Cases
### Compare Models Side-by-Side
```bash
# First, test the pretrained model
python evaluate_model.py --pretrained > results_pretrained.txt
# Then, test the finetuned model
python evaluate_model.py > results_finetuned.txt
# Compare the outputs
diff results_pretrained.txt results_finetuned.txt
```
### Creative vs Deterministic Generation
```bash
# Deterministic (greedy decoding) - same output every time
python evaluate_model.py
# Creative (sampling) - different output each time
python evaluate_model.py --use_sampling --temperature 0.7
# Very creative (higher temperature)
python evaluate_model.py --use_sampling --temperature 1.0
# More focused (lower temperature)
python evaluate_model.py --use_sampling --temperature 0.3
```
### Long-Form Generation
```bash
# Generate longer responses
python evaluate_model.py --max_new_tokens 500
```
## Test Cases
### Evaluation Script (evaluate_model.py)
Runs 6 test cases on a single model:
1. **Korean Wikipedia Article (Artificial Intelligence)** - Tests encyclopedic writing in Korean
2. **English Wikipedia Article (Artificial Intelligence)** - Ensures English preservation
3. **Korean Instruction (Explain Kimchi)** - Tests instruction-following for cultural topics
4. **English Instruction (Explain Thanksgiving Turkey)** - Tests English instruction-following
5. **Korean Instruction (Introduce Seoul)** - Tests factual knowledge in Korean
6. **Korean Instruction (Explain K-pop)** - Tests modern cultural knowledge
### Comparison Script (compare_models.py)
Runs 5 test cases across 3 models (15 total outputs):
1. **Korean Wikipedia - AI** - Shows Korean capability progression
2. **English Wikipedia - AI** - Validates English preservation (encyclopedic writing)
3. **Korean Instruction - Kimchi** - Shows instruction-following improvement
4. **Korean Instruction - Seoul** - Tests factual accuracy improvement
5. **English Instruction - Thanksgiving** - Validates English preservation (instruction-following)
The comparison script includes both English Wikipedia AND English Instruction tests to comprehensively validate that English capabilities remain strong throughout all training stages.
## Understanding the Output
### Color Coding
- 🔵 **Blue**: Loading and setup information
- 🟡 **Yellow**: Parameters and configuration
- 🟢 **Green**: Successful operations and output
- 🔴 **Red**: Errors
- 🔵 **Cyan**: Prompts and tips
### Evaluation Metrics (Manual)
When evaluating outputs, consider:
1. **Fluency**: Is the Korean grammatically correct?
2. **Factual Accuracy**: Are the facts correct?
3. **Instruction Following**: Does it answer the question?
4. **Coherence**: Does it make logical sense?
5. **Cultural Appropriateness**: Is cultural information accurate?
## Troubleshooting
### "Model path does not exist"
Make sure you've run `continued-pretrain.py` first to train and save the models.
### Out of Memory
Try:
```bash
# Use 4-bit quantization
python evaluate_model.py --load_in_4bit
# Reduce max sequence length
python evaluate_model.py --max_seq_length 1024
# Generate fewer tokens
python evaluate_model.py --max_new_tokens 100
```
### Outputs Too Short
Increase max tokens:
```bash
python evaluate_model.py --max_new_tokens 300
```
### Want Different Outputs Each Time
Enable sampling:
```bash
python evaluate_model.py --use_sampling
```
## Tips for Best Results
1. **Start with defaults**: Run with no arguments first
2. **Compare stages**: Test both `--pretrained` and final model
3. **Use sampling for variety**: Add `--use_sampling` for creative outputs
4. **Monitor GPU memory**: Check the memory stats in output
## Expected Performance
### Baseline Model (No Training)
- ❌ Korean: Poor, repetitive, often nonsensical
- ✅ English: Good, coherent, accurate
### Pretrained Model (After Korean Training)
- ⚠️ Korean: Improved fluency, better vocabulary
- ✅ English: Maintained quality
- ⚠️ Instructions: Better than baseline, but not perfect
### Finetuned Model (After SFT)
- ✅ Korean: Fluent, accurate, follows instructions
- ✅ English: Maintained quality
- ✅ Instructions: Good instruction-following in both languages
## Advanced Usage
### Batch Testing Multiple Configurations
Create a shell script:
```bash
#!/bin/bash
# test_configs.sh
echo "Testing different temperatures..."
for temp in 0.3 0.7 1.0; do
echo "=== Testing temperature=$temp ==="
python evaluate_model.py --use_sampling --temperature $temp \
--max_new_tokens 150 > results_temp_${temp}.txt
done
echo "Testing different token lengths..."
for tokens in 100 200 300; do
echo "=== Testing max_new_tokens=$tokens ==="
python evaluate_model.py --max_new_tokens $tokens \
> results_tokens_${tokens}.txt
done
```
### Custom Test Prompts
Modify the `run_evaluation()` function in `evaluate_model.py` to add your own test cases.
## References
- Main training script: `continued-pretrain.py`
- Unsloth documentation: https://docs.unsloth.ai
- Generation parameters: https://huggingface.co/docs/transformers/main_classes/text_generation
## Support
If you encounter issues:
1. Check that training completed successfully
2. Verify model files exist in `lora_model/` or `lora_model_pretrained/`
3. Ensure you have sufficient GPU memory
4. Try reducing `--max_seq_length` or `--max_new_tokens`
chapter8/continued-pretraining/compare_models.py
+296
View File
@@ -0,0 +1,296 @@
# -*- coding: utf-8 -*-
"""
Compare baseline → pretrained → finetuned Korean Mistral models (3-way comparison)
Shows progression from original model to final Korean-capable model
"""
import argparse
# 说明：unsloth / torch 等重型依赖在函数内按需导入，
# 这样 `python compare_models.py --help` 无需 GPU 环境即可查看参数。
# ANSI color codes for colored output
class Colors:
HEADER = '\033[95m'
BLUE = '\033[94m'
CYAN = '\033[96m'
GREEN = '\033[92m'
YELLOW = '\033[93m'
RED = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'
def print_section(title, color=Colors.CYAN):
"""Print a colored section header"""
print(f"\n{color}{Colors.BOLD}{'='*80}")
print(f"{title}")
print(f"{'='*80}{Colors.ENDC}\n")
def load_baseline_model(base_model="unsloth/mistral-7b-v0.3", max_seq_length=2048):
"""Load the original Mistral model (before any training)"""
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=base_model,
max_seq_length=max_seq_length,
dtype=None,
load_in_4bit=True,
)
FastLanguageModel.for_inference(model)
return model, tokenizer
def load_model(model_path, max_seq_length=2048):
"""Load a trained LoRA model"""
from unsloth import FastLanguageModel
model, tokenizer = FastLanguageModel.from_pretrained(
model_name=model_path,
max_seq_length=max_seq_length,
dtype=None,
load_in_4bit=True,
)
FastLanguageModel.for_inference(model)
return model, tokenizer
def generate_text(model, tokenizer, prompt, max_new_tokens=150, temperature=0.3):
"""Generate text without streaming"""
inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
outputs = model.generate(
**inputs,
max_new_tokens=max_new_tokens,
use_cache=True,
do_sample=True,
temperature=temperature,
pad_token_id=tokenizer.eos_token_id,
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
# Remove the prompt from the output
response = generated_text[len(prompt):].strip()
return response
def compare_on_prompt(baseline_model, baseline_tokenizer,
pretrained_model, pretrained_tokenizer,
finetuned_model, finetuned_tokenizer,
prompt, test_name, prompt_translation=None,
max_new_tokens=150, temperature=0.3):
"""Compare three models on the same prompt"""
print(f"\n{Colors.BOLD}{'='*80}")
print(f"{test_name}")
print(f"{'='*80}{Colors.ENDC}")
if prompt_translation:
print(f"{Colors.CYAN}Prompt (Translation): {prompt_translation}{Colors.ENDC}\n")
print(f"{Colors.YELLOW}Generating from BASELINE model (original Mistral)...{Colors.ENDC}")
baseline_output = generate_text(
baseline_model, baseline_tokenizer, prompt,
max_new_tokens, temperature
)
print(f"{Colors.YELLOW}Generating from PRETRAINED model (after Korean training)...{Colors.ENDC}")
pretrained_output = generate_text(
pretrained_model, pretrained_tokenizer, prompt,
max_new_tokens, temperature
)
print(f"{Colors.YELLOW}Generating from FINETUNED model (after instruction tuning)...{Colors.ENDC}")
finetuned_output = generate_text(
finetuned_model, finetuned_tokenizer, prompt,
max_new_tokens, temperature
)
# Display all three outputs
print(f"\n{Colors.RED}┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐{Colors.ENDC}")
print(f"{Colors.RED}│{Colors.ENDC}")
for line in baseline_output.split('\n'):
print(f"{Colors.RED}│{Colors.ENDC} {line}")
print(f"{Colors.RED}│{Colors.ENDC}")
print(f"{Colors.RED}└────────────────────────────────────────────────────────────────────┘{Colors.ENDC}\n")
print(f"{Colors.GREEN}┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐{Colors.ENDC}")
print(f"{Colors.GREEN}│{Colors.ENDC}")
for line in pretrained_output.split('\n'):
print(f"{Colors.GREEN}│{Colors.ENDC} {line}")
print(f"{Colors.GREEN}│{Colors.ENDC}")
print(f"{Colors.GREEN}└────────────────────────────────────────────────────────────────────┘{Colors.ENDC}\n")
print(f"{Colors.CYAN}┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐{Colors.ENDC}")
print(f"{Colors.CYAN}│{Colors.ENDC}")
for line in finetuned_output.split('\n'):
print(f"{Colors.CYAN}│{Colors.ENDC} {line}")
print(f"{Colors.CYAN}│{Colors.ENDC}")
print(f"{Colors.CYAN}└────────────────────────────────────────────────────────────────────┘{Colors.ENDC}\n")
def parse_args():
parser = argparse.ArgumentParser(
description="对比韩语 Mistral 的三个阶段模型：基础模型 → 继续预训练 → 指令微调。"
"在同一批中韩英提示上并排生成，直观展示韩语能力的提升与英语能力的保留。",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
parser.add_argument("--base_model", type=str, default="unsloth/mistral-7b-v0.3",
help="基础（未训练）模型名称")
parser.add_argument("--pretrained_path", type=str, default="lora_model_pretrained",
help="继续预训练后保存的 LoRA 模型目录")
parser.add_argument("--finetuned_path", type=str, default="lora_model",
help="指令微调后保存的最终 LoRA 模型目录")
parser.add_argument("--max_seq_length", type=int, default=2048,
help="最大序列长度")
parser.add_argument("--max_new_tokens", type=int, default=150,
help="每次生成的最大 token 数")
parser.add_argument("--temperature", type=float, default=0.3,
help="采样温度（越低越确定）")
return parser.parse_args()
def main():
args = parse_args()
print_section("🔬 KOREAN MISTRAL 3-WAY MODEL COMPARISON", Colors.HEADER)
print(f"{Colors.YELLOW}This script compares three model stages:{Colors.ENDC}")
print(f" 1. {Colors.RED}Baseline{Colors.ENDC} - Original Mistral (no Korean training)")
print(f" 2. {Colors.GREEN}Pretrained{Colors.ENDC} - After Korean Wikipedia training")
print(f" 3. {Colors.CYAN}Finetuned{Colors.ENDC} - After instruction tuning")
print(f"\n{Colors.CYAN}Generation settings: temperature={args.temperature}, do_sample=True (no repetition_penalty){Colors.ENDC}\n")
# Load all three models
print_section("📥 LOADING MODELS", Colors.BLUE)
print(f"{Colors.YELLOW}Loading baseline model (original Mistral v0.3)...{Colors.ENDC}")
baseline_model, baseline_tokenizer = load_baseline_model(args.base_model, args.max_seq_length)
print(f"{Colors.GREEN}✓ Baseline model loaded{Colors.ENDC}")
print(f"\n{Colors.YELLOW}Loading pretrained model (after Korean pretraining)...{Colors.ENDC}")
pretrained_model, pretrained_tokenizer = load_model(args.pretrained_path, args.max_seq_length)
print(f"{Colors.GREEN}✓ Pretrained model loaded{Colors.ENDC}")
print(f"\n{Colors.YELLOW}Loading finetuned model (after instruction tuning)...{Colors.ENDC}")
finetuned_model, finetuned_tokenizer = load_model(args.finetuned_path, args.max_seq_length)
print(f"{Colors.GREEN}✓ Finetuned model loaded{Colors.ENDC}")
# Define prompts
wikipedia_prompt_korean = """위키피디아 기사
### 제목: {}
### 기사:
{}"""
wikipedia_prompt_english = """Wikipedia Article
### Title: {}
### Article:
{}"""
alpaca_prompt_korean = """다음은 작업을 설명하는 명령입니다. 요청을 적절하게 완료하는 응답을 작성하세요.
### 지침:
{}
### 응답:
{}"""
alpaca_prompt_english = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{}
### Response:
{}"""
print_section("🧪 RUNNING 3-WAY COMPARISONS", Colors.CYAN)
# Test 1: Korean Wikipedia
compare_on_prompt(
baseline_model, baseline_tokenizer,
pretrained_model, pretrained_tokenizer,
finetuned_model, finetuned_tokenizer,
wikipedia_prompt_korean.format("인공지능", ""),
"Test 1: Korean Wikipedia - Artificial Intelligence (인공지능)",
"Wikipedia Article / Title: Artificial Intelligence / Article:",
max_new_tokens=args.max_new_tokens, temperature=args.temperature
)
# Test 2: English Wikipedia - Preservation Check
compare_on_prompt(
baseline_model, baseline_tokenizer,
pretrained_model, pretrained_tokenizer,
finetuned_model, finetuned_tokenizer,
wikipedia_prompt_english.format("Artificial Intelligence", ""),
"Test 2: English Wikipedia - Artificial Intelligence (Preservation Check)",
None,
max_new_tokens=args.max_new_tokens, temperature=args.temperature
)
# Test 3: Korean Instruction - Kimchi
compare_on_prompt(
baseline_model, baseline_tokenizer,
pretrained_model, pretrained_tokenizer,
finetuned_model, finetuned_tokenizer,
alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", ""),
"Test 3: Korean Instruction - Explain Kimchi",
"Instruction: Explain about kimchi, a traditional Korean food. / Response:",
max_new_tokens=args.max_new_tokens, temperature=args.temperature
)
# Test 4: Korean Instruction - Seoul
compare_on_prompt(
baseline_model, baseline_tokenizer,
pretrained_model, pretrained_tokenizer,
finetuned_model, finetuned_tokenizer,
alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", ""),
"Test 4: Korean Instruction - Introduce Seoul",
"Instruction: Briefly introduce Seoul, the capital of South Korea. / Response:",
max_new_tokens=args.max_new_tokens, temperature=args.temperature
)
# Test 5: English Instruction - Preservation Check
compare_on_prompt(
baseline_model, baseline_tokenizer,
pretrained_model, pretrained_tokenizer,
finetuned_model, finetuned_tokenizer,
alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", ""),
"Test 5: English Instruction - Thanksgiving Turkey (Preservation Check)",
None,
max_new_tokens=args.max_new_tokens, temperature=args.temperature
)
print_section("📊 COMPARISON COMPLETE", Colors.GREEN)
print(f"{Colors.CYAN}{'='*80}")
print(f"💡 What to Look For:")
print(f"{'='*80}{Colors.ENDC}")
print(f"\n{Colors.RED}Baseline Model (Red boxes - Original Mistral):{Colors.ENDC}")
print(f" • Korean: Should be POOR - repetitive, nonsensical")
print(f" • English: Should be GOOD - this is the starting point")
print(f" • Shows what model knows BEFORE any Korean training")
print(f"\n{Colors.GREEN}Pretrained Model (Green boxes - After Korean Wikipedia):{Colors.ENDC}")
print(f" • Korean: Should show IMPROVED fluency and vocabulary")
print(f" • Better Korean sentence structure")
print(f" • Weak instruction-following (only learned language, not how to follow instructions)")
print(f" • English: Should REMAIN strong (no catastrophic forgetting)")
print(f"\n{Colors.CYAN}Finetuned Model (Cyan boxes - After Instruction Tuning):{Colors.ENDC}")
print(f" • Korean: Should be FLUENT with GOOD instruction-following")
print(f" • More structured and complete responses")
print(f" • Directly answers questions")
print(f" • English: Should REMAIN strong")
print(f"\n{Colors.YELLOW}Key Progression to Observe:{Colors.ENDC}")
print(f" 📊 Korean Quality: {Colors.RED}Poor{Colors.ENDC} → {Colors.GREEN}Better{Colors.ENDC} → {Colors.CYAN}Best{Colors.ENDC}")
print(f" 📊 Instruction: {Colors.RED}Weak{Colors.ENDC} → {Colors.GREEN}Weak{Colors.ENDC} → {Colors.CYAN}Strong{Colors.ENDC}")
print(f" 📊 English Quality: {Colors.RED}Good{Colors.ENDC} → {Colors.GREEN}Good{Colors.ENDC} → {Colors.CYAN}Good{Colors.ENDC}")
print(f" 📊 Repetition: {Colors.RED}High{Colors.ENDC} → {Colors.GREEN}Medium{Colors.ENDC} → {Colors.CYAN}Low{Colors.ENDC}")
print(f"\n{Colors.YELLOW}This demonstrates:{Colors.ENDC}")
print(f" ✓ Continued pretraining successfully teaches new language (Korean)")
print(f" ✓ Instruction tuning teaches how to follow instructions in the new language")
print(f" ✓ English capability is preserved throughout (no catastrophic forgetting)")
print(f" ✓ Both Wikipedia and Instruction tasks show English preservation")
print(f" ✓ Two-stage approach is necessary: language first, then instruction-following")
print(f"\n{Colors.CYAN}💡 Note: Compare the English tests (Tests 2 & 5) across all three models.")
print(f"All three should perform similarly well, proving no English degradation.{Colors.ENDC}")
print()
if __name__ == "__main__":
main()
chapter8/continued-pretraining/continued-pretrain.py
+645
View File
@@ -0,0 +1,645 @@
# -*- coding: utf-8 -*-
"""Continued pretraining - Korean + Unsloth.ipynb
Automatically generated by Colab.
Original file is located at
https://colab.research.google.com/drive/1tEd1FrOXWMnCU9UIvdYhs61tkxdMuKZu
To run this, press "*Runtime*" and press "*Run all*" on a **free** Tesla T4 Google Colab instance!
<div class="align-center">
<a href="https://github.com/unslothai/unsloth"><img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="115"></a>
<a href="https://discord.gg/u54VK8m8tk"><img src="https://github.com/unslothai/unsloth/raw/main/images/Discord button.png" width="145"></a>
<a href="https://ko-fi.com/unsloth"><img src="https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png" width="145"></a></a> Join Discord if you need help + ⭐ <i>Star us on <a href="https://github.com/unslothai/unsloth">Github</a> </i> ⭐
</div>
To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://github.com/unslothai/unsloth#installation-instructions---conda).
You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save) (eg for Llama.cpp).
We will use the Korean subset of the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia) to first continually pretrain Mistral v3, then use the [Alpaca GPT4 Dataset](https://huggingface.co/datasets/FreedomIntelligence/alpaca-gpt4-korean) translated into Korean to further finetune the model to let it follow instructions in Korean.
"""
# ============================================================================
# IMPORTS
# ============================================================================
import os
import argparse
# ============================================================================
# 命令行参数（默认值与原始脚本硬编码值完全一致，保证行为不变）
# ============================================================================
def parse_args():
parser = argparse.ArgumentParser(
description="韩语 Mistral 继续预训练 + 指令微调（Unsloth / LoRA）。"
"先用韩语维基百科做继续预训练注入韩语能力，再用韩语 Alpaca 数据做 SFT。",
formatter_class=argparse.ArgumentDefaultsHelpFormatter,
)
# 基础模型
parser.add_argument("--base_model", type=str, default="unsloth/mistral-7b-v0.3",
help="基础模型名称（HuggingFace / Unsloth 仓库名）")
parser.add_argument("--max_seq_len", type=int, default=2048,
help="最大序列长度")
parser.add_argument("--no_4bit", action="store_true",
help="关闭 4bit 量化加载（默认开启 4bit 以节省显存）")
# LoRA 配置
parser.add_argument("--lora_rank", type=int, default=128,
help="LoRA 秩 r（继续预训练建议较大，如 128）")
parser.add_argument("--lora_alpha", type=int, default=32,
help="LoRA alpha")
# 数据集
parser.add_argument("--wiki_dataset", type=str, default="wikimedia/wikipedia",
help="继续预训练用的维基百科数据集")
parser.add_argument("--wiki_config", type=str, default="20231101.ko",
help="维基百科数据集的语言/版本配置（默认韩语快照）")
parser.add_argument("--wiki_train_size", type=float, default=0.05,
help="维基百科数据集抽样比例（0~1，默认取 5%% 以加速训练）")
parser.add_argument("--alpaca_dataset", type=str,
default="FreedomIntelligence/alpaca-gpt4-korean",
help="指令微调（SFT）用的韩语 Alpaca 数据集")
# 训练超参数
parser.add_argument("--pretrain_epochs", type=int, default=1,
help="继续预训练阶段的训练轮数")
parser.add_argument("--pretrain_max_steps", type=int, default=-1,
help="继续预训练最大步数（-1 表示不限制，按 epoch 训练）")
parser.add_argument("--sft_epochs", type=int, default=2,
help="指令微调阶段的训练轮数")
parser.add_argument("--sft_max_steps", type=int, default=-1,
help="指令微调最大步数（-1 表示不限制，按 epoch 训练）")
# 输出目录
parser.add_argument("--pretrain_output_dir", type=str, default="outputs_pretrain",
help="继续预训练的检查点目录")
parser.add_argument("--sft_output_dir", type=str, default="outputs_sft",
help="指令微调的检查点目录")
parser.add_argument("--pretrained_save_dir", type=str, default="lora_model_pretrained",
help="继续预训练后保存的 LoRA 模型目录")
parser.add_argument("--final_save_dir", type=str, default="lora_model",
help="指令微调后保存的最终 LoRA 模型目录")
return parser.parse_args()
# 先解析参数（放在重型导入之前，这样 `--help` 无需 GPU / Unsloth 也能运行）
args = parse_args()
import torch
from unsloth import FastLanguageModel, is_bfloat16_supported, UnslothTrainer, UnslothTrainingArguments
from transformers import TrainingArguments, TextStreamer
from datasets import load_dataset
# ANSI color codes for colored output
class Colors:
HEADER = '\033[95m'
BLUE = '\033[94m'
CYAN = '\033[96m'
GREEN = '\033[92m'
YELLOW = '\033[93m'
RED = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'
def print_section(title, color=Colors.CYAN):
"""Print a colored section header"""
print(f"\n{color}{Colors.BOLD}{'='*70}")
print(f"{title}")
print(f"{'='*70}{Colors.ENDC}\n")
# ============================================================================
# MODEL SETUP
# ============================================================================
print_section("🚀 LOADING MODEL", Colors.BLUE)
max_seq_length = args.max_seq_len # Choose any! We auto support RoPE Scaling internally!
dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
load_in_4bit = not args.no_4bit # Use 4bit quantization to reduce memory usage. Can be False.
# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
fourbit_models = [
"unsloth/mistral-7b-v0.3-bnb-4bit", # New Mistral v3 2x faster!
"unsloth/mistral-7b-instruct-v0.3-bnb-4bit",
"unsloth/llama-3-8b-bnb-4bit", # Llama-3 15 trillion tokens model 2x faster!
"unsloth/llama-3-8b-Instruct-bnb-4bit",
"unsloth/llama-3-70b-bnb-4bit",
"unsloth/Phi-3-mini-4k-instruct", # Phi-3 2x faster!
"unsloth/Phi-3-medium-4k-instruct",
"unsloth/mistral-7b-bnb-4bit",
"unsloth/gemma-7b-bnb-4bit", # Gemma 2.2x faster!
] # More models at https://huggingface.co/unsloth
print(f"{Colors.GREEN}Loading {args.base_model}...{Colors.ENDC}")
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = args.base_model, # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
# token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
)
print_section("⚙️ ADDING LORA ADAPTERS", Colors.BLUE)
print(f"{Colors.GREEN}Adding LoRA adapters - only updating 1-10% of parameters!")
print(f"Including embed_tokens and lm_head for continual pretraining{Colors.ENDC}")
model = FastLanguageModel.get_peft_model(
model,
r = args.lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
"embed_tokens", "lm_head",], # Add for continual pretraining
lora_alpha = args.lora_alpha,
lora_dropout = 0, # Supports any, but = 0 is optimized
bias = "none", # Supports any, but = "none" is optimized
# [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
random_state = 3407,
use_rslora = True, # We support rank stabilized LoRA
loftq_config = None, # And LoftQ
)
# ============================================================================
# TESTING ORIGINAL MODEL (BASELINE)
# ============================================================================
print_section("🧪 TESTING ORIGINAL MODEL (BASELINE)", Colors.CYAN)
print(f"{Colors.YELLOW}Testing the base Mistral model BEFORE any training")
print(f"This establishes baseline for Korean and English capabilities{Colors.ENDC}")
FastLanguageModel.for_inference(model)
text_streamer = TextStreamer(tokenizer)
# Prepare prompts (define them early)
_wikipedia_prompt = """Wikipedia Article
### Title: {}
### Article:
{}"""
wikipedia_prompt_korean = """위키피디아 기사
### 제목: {}
### 기사:
{}"""
_alpaca_prompt_english = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{}
### Response:
{}"""
alpaca_prompt_korean = """다음은 작업을 설명하는 명령입니다. 요청을 적절하게 완료하는 응답을 작성하세요.
### 지침:
{}
### 응답:
{}"""
# Test 1: Korean Wikipedia (baseline - should be poor)
print(f"\n{Colors.BOLD}Test 1: Korean Wikipedia Article (인공지능){Colors.ENDC}")
print("="*70)
test_prompt = wikipedia_prompt_korean.format("인공지능", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 2: English Wikipedia (baseline - should be good)
print(f"\n{Colors.BOLD}Test 2: English Wikipedia Article (Artificial Intelligence){Colors.ENDC}")
print("="*70)
test_prompt = _wikipedia_prompt.format("Artificial Intelligence", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 3: Korean Instruction (baseline - should be poor)
print(f"\n{Colors.BOLD}Test 3: Korean Instruction (Korean Culture){Colors.ENDC}")
print("="*70)
test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 4: English Instruction (baseline - should be good)
print(f"\n{Colors.BOLD}Test 4: English Instruction (American Culture){Colors.ENDC}")
print("="*70)
test_prompt = _alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 5: Korean Instruction - Seoul (baseline - should be poor)
print(f"\n{Colors.BOLD}Test 5: Korean Instruction (Korean Geography){Colors.ENDC}")
print("="*70)
test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
print(f"{Colors.GREEN}✓ Baseline testing complete. Original model should be good at English but poor at Korean.{Colors.ENDC}\n")
# ============================================================================
# DATA PREPARATION
# ============================================================================
print_section("📚 DATA PREPARATION - WIKIPEDIA KOREAN DATASET", Colors.CYAN)
print(f"{Colors.YELLOW}Loading Korean Wikipedia dataset...")
print(f"We'll use 5% of the dataset to speed up training{Colors.ENDC}")
# Use the prompts already defined above
EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN
def formatting_prompts_func_wiki(examples):
titles = examples["title"]
texts = examples["text"]
outputs = []
for title, text in zip(titles, texts):
# Must add EOS_TOKEN, otherwise your generation will go on forever!
text = wikipedia_prompt_korean.format(title, text) + EOS_TOKEN
outputs.append(text)
return { "text" : outputs, }
dataset = load_dataset(args.wiki_dataset, args.wiki_config, split = "train",)
# We select 5% of the data to make training faster!
dataset = dataset.train_test_split(train_size = args.wiki_train_size)["train"]
dataset = dataset.map(formatting_prompts_func_wiki, batched = True,)
print(f"{Colors.GREEN}✓ Wikipedia dataset loaded: {len(dataset)} examples{Colors.ENDC}")
print_section("📚 DATA PREPARATION - ALPACA KOREAN DATASET", Colors.CYAN)
print(f"{Colors.YELLOW}Loading Alpaca GPT4 Korean dataset for instruction finetuning...{Colors.ENDC}")
alpaca_dataset = load_dataset(args.alpaca_dataset, split = "train")
# Use the prompts already defined above
def formatting_prompts_func_alpaca(conversations):
texts = []
conversations = conversations["conversations"]
for convo in conversations:
# Must add EOS_TOKEN, otherwise your generation will go on forever!
text = alpaca_prompt_korean.format(convo[0]["value"], convo[1]["value"]) + EOS_TOKEN
texts.append(text)
return { "text" : texts, }
alpaca_dataset = alpaca_dataset.map(formatting_prompts_func_alpaca, batched = True,)
print(f"{Colors.GREEN}✓ Alpaca dataset loaded: {len(alpaca_dataset)} examples{Colors.ENDC}")
print(f"\nExample from Alpaca dataset:")
print(alpaca_dataset[0])
# ============================================================================
# CONTINUED PRETRAINING
# ============================================================================
print_section("🎯 CONTINUED PRETRAINING ON KOREAN WIKIPEDIA", Colors.GREEN)
print(f"{Colors.YELLOW}Training the model on Korean Wikipedia to learn the language...")
print(f"Using embedding_learning_rate (1e-5) smaller than learning_rate (5e-5)")
print(f"💾 Checkpoints will be saved every 100 steps to: outputs_pretrain/")
print(f"Only the 5 most recent checkpoints will be kept{Colors.ENDC}")
# Set WandB project for pretraining
os.environ["WANDB_PROJECT"] = "unsloth-continued-pretraining"
trainer = UnslothTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 2,
args = UnslothTrainingArguments(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 8,
# Use warmup_ratio and num_train_epochs for longer runs!
max_steps = args.pretrain_max_steps,
warmup_steps = 10,
warmup_ratio = 0.1,
num_train_epochs = args.pretrain_epochs,
# Select a 2 to 10x smaller learning rate for the embedding matrices!
learning_rate = 5e-5,
embedding_learning_rate = 1e-5,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.01,
lr_scheduler_type = "linear",
seed = 42,
output_dir = args.pretrain_output_dir,
# Checkpoint saving
save_strategy = "steps",
save_steps = 100,
# Keep only the 5 most recent checkpoints, as the banner above
# promises. Without this HF keeps every one, and a ~2000-step run
# writes ~20 LoRA+optimizer checkpoints (fills a free-Colab disk).
save_total_limit = 5,
# WandB configuration
report_to = "wandb",
run_name = "korean-mistral-pretrain",
),
)
print_section("💾 MEMORY STATS - BEFORE PRETRAINING", Colors.YELLOW)
gpu_stats = torch.cuda.get_device_properties(0)
start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.")
print(f"{start_gpu_memory} GB of memory reserved.")
trainer_stats = trainer.train()
# Finish wandb run for pretraining
import wandb
if wandb.run is not None:
wandb.finish()
print(f"{Colors.CYAN}✓ Finished wandb run for pretraining{Colors.ENDC}")
print_section("💾 SAVING PRETRAINED MODEL", Colors.GREEN)
model.save_pretrained(args.pretrained_save_dir) # Local saving
tokenizer.save_pretrained(args.pretrained_save_dir)
print(f"{Colors.GREEN}✓ Model saved to: {args.pretrained_save_dir}/{Colors.ENDC}")
# ============================================================================
# TESTING PRETRAINED MODEL
# ============================================================================
print_section("🧪 TESTING PRETRAINED MODEL (AFTER KOREAN PRETRAINING)", Colors.CYAN)
print(f"{Colors.YELLOW}Testing after Korean pretraining - before instruction finetuning")
print(f"Korean should improve, English should remain strong{Colors.ENDC}")
# Test the pretrained model before instruction finetuning
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
text_streamer = TextStreamer(tokenizer)
# Test 1: Korean Wikipedia (same as baseline)
print(f"\n{Colors.BOLD}Test 1: Korean Wikipedia Article (인공지능){Colors.ENDC}")
print("="*70)
test_prompt = wikipedia_prompt_korean.format("인공지능", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 2: English Wikipedia (same as baseline)
print(f"\n{Colors.BOLD}Test 2: English Wikipedia Article (Artificial Intelligence){Colors.ENDC}")
print("="*70)
test_prompt = _wikipedia_prompt.format("Artificial Intelligence", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 3: Korean Instruction (same as baseline - should improve but not follow perfectly yet)
print(f"\n{Colors.BOLD}Test 3: Korean Instruction (Korean Culture){Colors.ENDC}")
print("="*70)
test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 4: English Instruction (same as baseline)
print(f"\n{Colors.BOLD}Test 4: English Instruction (American Culture){Colors.ENDC}")
print("="*70)
test_prompt = _alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 5: Korean Instruction - Seoul (should improve but not perfect yet)
print(f"\n{Colors.BOLD}Test 5: Korean Instruction (Korean Geography){Colors.ENDC}")
print("="*70)
test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
print(f"{Colors.GREEN}✓ Pretrained model testing complete.")
print(f"{Colors.CYAN}💡 Korean should be significantly improved, English should remain strong.")
print(f"Instruction following may improve but not perfect yet - that's what SFT is for.{Colors.ENDC}\n")
# ============================================================================
# INSTRUCTION FINETUNING
# ============================================================================
print_section("🎓 INSTRUCTION FINETUNING ON ALPACA KOREAN", Colors.GREEN)
print(f"{Colors.YELLOW}Now finetuning the model to follow Korean instructions...")
print(f"Using the Alpaca GPT4 dataset translated to Korean")
print(f"💾 Checkpoints will be saved every 100 steps to: outputs_sft/")
print(f"Only the 5 most recent checkpoints will be kept{Colors.ENDC}")
# Set WandB project for finetuning
os.environ["WANDB_PROJECT"] = "unsloth-continued-finetuning"
trainer = UnslothTrainer(
model = model,
tokenizer = tokenizer,
train_dataset = alpaca_dataset,
dataset_text_field = "text",
max_seq_length = max_seq_length,
dataset_num_proc = 8,
args = UnslothTrainingArguments(
per_device_train_batch_size = 2,
gradient_accumulation_steps = 8,
# Use num_train_epochs and warmup_ratio for longer runs!
max_steps = args.sft_max_steps,
warmup_steps = 10,
warmup_ratio = 0.1,
num_train_epochs = args.sft_epochs,
# Select a 2 to 10x smaller learning rate for the embedding matrices!
learning_rate = 5e-5,
embedding_learning_rate = 1e-5,
fp16 = not is_bfloat16_supported(),
bf16 = is_bfloat16_supported(),
logging_steps = 1,
optim = "adamw_8bit",
weight_decay = 0.00,
lr_scheduler_type = "linear",
seed = 42,
output_dir = args.sft_output_dir,
# Checkpoint saving
save_strategy = "steps",
save_steps = 100,
# Keep only the 5 most recent checkpoints, as the banner above
# promises. Without this HF keeps every one, and a ~2000-step run
# writes ~20 LoRA+optimizer checkpoints (fills a free-Colab disk).
save_total_limit = 5,
# WandB configuration
report_to = "wandb",
run_name = "korean-mistral-finetune",
),
)
trainer_stats = trainer.train()
# Finish wandb run for SFT
if wandb.run is not None:
wandb.finish()
print(f"{Colors.CYAN}✓ Finished wandb run for SFT{Colors.ENDC}")
print_section("📊 FINAL MEMORY AND TIME STATS", Colors.YELLOW)
used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
used_memory_for_lora = round(used_memory - start_gpu_memory, 3)
used_percentage = round(used_memory /max_memory*100, 3)
lora_percentage = round(used_memory_for_lora/max_memory*100, 3)
print(f"{trainer_stats.metrics['train_runtime']} seconds used for training.")
print(f"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.")
print(f"Peak reserved memory = {used_memory} GB.")
print(f"Peak reserved memory for training = {used_memory_for_lora} GB.")
print(f"Peak reserved memory % of max memory = {used_percentage} %.")
print(f"Peak reserved memory for training % of max memory = {lora_percentage} %.")
# ============================================================================
# INFERENCE - TESTING FINETUNED MODEL
# ============================================================================
print_section("🎯 INFERENCE - TESTING FINETUNED MODEL", Colors.CYAN)
print(f"{Colors.YELLOW}Testing the instruction-finetuned model - should follow instructions in Korean AND English{Colors.ENDC}")
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
text_streamer = TextStreamer(tokenizer)
# Test 1: Korean Wikipedia (same as baseline)
print(f"\n{Colors.BOLD}Test 1: Korean Wikipedia Article (인공지능){Colors.ENDC}")
print("="*70)
test_prompt = wikipedia_prompt_korean.format("인공지능", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 2: English Wikipedia (same as baseline)
print(f"\n{Colors.BOLD}Test 2: English Wikipedia Article (Artificial Intelligence){Colors.ENDC}")
print("="*70)
test_prompt = _wikipedia_prompt.format("Artificial Intelligence", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 3: Korean Instruction (same as baseline - should follow WELL now)
print(f"\n{Colors.BOLD}Test 3: Korean Instruction (Korean Culture){Colors.ENDC}")
print("="*70)
test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 4: English Instruction (same as baseline)
print(f"\n{Colors.BOLD}Test 4: English Instruction (American Culture){Colors.ENDC}")
print("="*70)
test_prompt = _alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
# Test 5: Korean Instruction - Seoul (should follow WELL now)
print(f"\n{Colors.BOLD}Test 5: Korean Instruction (Korean Geography){Colors.ENDC}")
print("="*70)
test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
print("="*70 + "\n")
print(f"{Colors.GREEN}✓ Finetuned model testing complete!")
print(f"{Colors.CYAN}💡 Model should now follow instructions well in both Korean AND English.{Colors.ENDC}\n")
# ============================================================================
# SAVING FINAL MODEL
# ============================================================================
print_section("💾 SAVING FINAL FINETUNED MODEL", Colors.GREEN)
model.save_pretrained(args.final_save_dir) # Local saving
tokenizer.save_pretrained(args.final_save_dir)
print(f"{Colors.GREEN}✓ Final model saved to: {args.final_save_dir}/{Colors.ENDC}")
# model.push_to_hub("your_name/lora_model", token = "...") # Online saving
# tokenizer.push_to_hub("your_name/lora_model", token = "...") # Online saving
print(f"\n{Colors.CYAN}Note: This only saves LoRA adapters, not the full model.")
print(f"For 16bit or GGUF formats, see the export options below.{Colors.ENDC}")
# ============================================================================
# LOADING SAVED MODEL (OPTIONAL)
# ============================================================================
"""
print_section("📥 LOADING SAVED MODEL", Colors.BLUE)
# Set to True to test loading
if False:
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
inputs = tokenizer(
[
alpaca_prompt.format(
# "Describe the planet Earth extensively.", # instruction
"지구를 광범위하게 설명하세요.",
"", # output - leave this blank for generation!
),
], return_tensors = "pt").to("cuda")
text_streamer = TextStreamer(tokenizer)
_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 256)
"""
# ============================================================================
# EXPORT OPTIONS
# ============================================================================
print_section("📦 EXPORT OPTIONS", Colors.BLUE)
print(f"{Colors.YELLOW}Various export formats available (currently disabled):{Colors.ENDC}")
print("• Float16 merged model")
print("• 4bit merged model")
print("• LoRA adapters only")
print("• GGUF format for llama.cpp")
# Merge to 16bit
if False:
print_section("💾 EXPORTING TO FLOAT16", Colors.GREEN)
model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",)
# model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_16bit", token = "")
# Merge to 4bit
if False:
print_section("💾 EXPORTING TO 4BIT", Colors.GREEN)
model.save_pretrained_merged("model", tokenizer, save_method = "merged_4bit",)
# model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_4bit", token = "")
# Just LoRA adapters
if False:
print_section("💾 EXPORTING LORA ADAPTERS", Colors.GREEN)
model.save_pretrained_merged("model", tokenizer, save_method = "lora",)
# model.push_to_hub_merged("hf/model", tokenizer, save_method = "lora", token = "")
# GGUF exports
if False:
print_section("💾 EXPORTING TO GGUF Q8_0", Colors.GREEN)
model.save_pretrained_gguf("model", tokenizer,)
# model.push_to_hub_gguf("hf/model", tokenizer, token = "")
if False:
print_section("💾 EXPORTING TO GGUF F16", Colors.GREEN)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "f16")
# model.push_to_hub_gguf("hf/model", tokenizer, quantization_method = "f16", token = "")
if False:
print_section("💾 EXPORTING TO GGUF Q4_K_M", Colors.GREEN)
model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")
# model.push_to_hub_gguf("hf/model", tokenizer, quantization_method = "q4_k_m", token = "")
print_section("✅ TRAINING COMPLETE!", Colors.GREEN)
print(f"{Colors.BOLD}Your Korean Mistral model is ready to use!{Colors.ENDC}")
print(f"\n{Colors.CYAN}For more information:{Colors.ENDC}")
print("• Discord: https://discord.gg/u54VK8m8tk")
print("• GitHub: https://github.com/unslothai/unsloth")
print("• Documentation: https://docs.unsloth.ai")
chapter8/continued-pretraining/evaluate_model.py
+285
View File
@@ -0,0 +1,285 @@
# -*- coding: utf-8 -*-
"""
Evaluation script for Korean Mistral continued-pretrained models
Loads saved LoRA adapters and evaluates on Korean and English tasks
"""
import os
import argparse
# 说明：unsloth / torch / transformers 等重型依赖在函数内按需导入，
# 这样 `python evaluate_model.py --help` 无需 GPU 环境即可查看参数。
# ANSI color codes for colored output
class Colors:
HEADER = '\033[95m'
BLUE = '\033[94m'
CYAN = '\033[96m'
GREEN = '\033[92m'
YELLOW = '\033[93m'
RED = '\033[91m'
ENDC = '\033[0m'
BOLD = '\033[1m'
UNDERLINE = '\033[4m'
def print_section(title, color=Colors.CYAN):
"""Print a colored section header"""
print(f"\n{color}{Colors.BOLD}{'='*70}")
print(f"{title}")
print(f"{'='*70}{Colors.ENDC}\n")
def load_model(model_path, max_seq_length=2048, dtype=None, load_in_4bit=True):
"""Load the saved LoRA model"""
from unsloth import FastLanguageModel
print_section(f"📥 LOADING MODEL FROM: {model_path}", Colors.BLUE)
print(f"{Colors.YELLOW}Loading model and tokenizer...{Colors.ENDC}")
model, tokenizer = FastLanguageModel.from_pretrained(
model_name = model_path,
max_seq_length = max_seq_length,
dtype = dtype,
load_in_4bit = load_in_4bit,
)
FastLanguageModel.for_inference(model) # Enable native 2x faster inference
print(f"{Colors.GREEN}✓ Model loaded successfully!{Colors.ENDC}")
return model, tokenizer
def run_evaluation(model, tokenizer, max_new_tokens=150,
temperature=0.7, top_p=0.9, use_sampling=False):
"""Run all evaluation tests"""
from transformers import TextStreamer
print_section("🧪 RUNNING EVALUATION TESTS", Colors.CYAN)
print(f"{Colors.YELLOW}Generation Parameters:{Colors.ENDC}")
print(f" • max_new_tokens: {max_new_tokens}")
if use_sampling:
print(f" • temperature: {temperature}")
print(f" • top_p: {top_p}")
print(f" • Sampling: Enabled")
else:
print(f" • Sampling: Disabled (greedy decoding)")
text_streamer = TextStreamer(tokenizer, skip_special_tokens=True)
# Define prompts
wikipedia_prompt_korean = """위키피디아 기사
### 제목: {}
### 기사:
{}"""
wikipedia_prompt_english = """Wikipedia Article
### Title: {}
### Article:
{}"""
alpaca_prompt_korean = """다음은 작업을 설명하는 명령입니다. 요청을 적절하게 완료하는 응답을 작성하세요.
### 지침:
{}
### 응답:
{}"""
alpaca_prompt_english = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
{}
### Response:
{}"""
# Prepare generation kwargs
gen_kwargs = {
"max_new_tokens": max_new_tokens,
"use_cache": True,
}
if use_sampling:
gen_kwargs.update({
"do_sample": True,
"temperature": temperature,
"top_p": top_p,
})
# Test 1: Korean Wikipedia Article
print(f"\n{Colors.BOLD}{'='*70}")
print(f"Test 1: Korean Wikipedia Article - Artificial Intelligence (인공지능)")
print(f"{'='*70}{Colors.ENDC}")
print(f"{Colors.CYAN}Prompt (Translation): Wikipedia Article / Title: Artificial Intelligence / Article:{Colors.ENDC}\n")
test_prompt = wikipedia_prompt_korean.format("인공지능", "")
inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
_ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
# Test 2: English Wikipedia Article
print(f"\n{Colors.BOLD}{'='*70}")
print(f"Test 2: English Wikipedia Article - Artificial Intelligence")
print(f"{'='*70}{Colors.ENDC}\n")
test_prompt = wikipedia_prompt_english.format("Artificial Intelligence", "")
inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
print(f"{Colors.GREEN}[ENGLISH OUTPUT]{Colors.ENDC}")
_ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
# Test 3: Korean Instruction (Kimchi)
print(f"\n{Colors.BOLD}{'='*70}")
print(f"Test 3: Korean Instruction - Explain about Kimchi")
print(f"{'='*70}{Colors.ENDC}")
print(f"{Colors.CYAN}Prompt (Translation): Instruction: Explain about kimchi, a traditional Korean food. / Response:{Colors.ENDC}\n")
test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
_ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
# Test 4: English Instruction (Thanksgiving)
print(f"\n{Colors.BOLD}{'='*70}")
print(f"Test 4: English Instruction - Explain about Thanksgiving Turkey")
print(f"{'='*70}{Colors.ENDC}\n")
test_prompt = alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
print(f"{Colors.GREEN}[ENGLISH OUTPUT]{Colors.ENDC}")
_ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
# Additional Korean tests
print(f"\n{Colors.BOLD}{'='*70}")
print(f"Test 5: Korean Instruction - Explain about Seoul")
print(f"{'='*70}{Colors.ENDC}")
print(f"{Colors.CYAN}Prompt (Translation): Instruction: Briefly introduce Seoul, the capital of South Korea. / Response:{Colors.ENDC}\n")
test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
_ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
# Test 6: Korean Instruction (K-pop)
print(f"\n{Colors.BOLD}{'='*70}")
print(f"Test 6: Korean Instruction - Explain about K-pop")
print(f"{'='*70}{Colors.ENDC}")
print(f"{Colors.CYAN}Prompt (Translation): Instruction: Explain what K-pop is. / Response:{Colors.ENDC}\n")
test_prompt = alpaca_prompt_korean.format("K-pop이 무엇인지 설명해주세요.", "")
inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
_ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
print_section("✅ EVALUATION COMPLETE", Colors.GREEN)
def main():
parser = argparse.ArgumentParser(description="Evaluate Korean Mistral LoRA models")
parser.add_argument(
"--model_path",
type=str,
default="lora_model",
help="Path to the saved LoRA model (default: lora_model)"
)
parser.add_argument(
"--pretrained",
action="store_true",
help="Load the pretrained model (before SFT) instead of final model"
)
parser.add_argument(
"--max_seq_length",
type=int,
default=2048,
help="Maximum sequence length (default: 2048)"
)
parser.add_argument(
"--load_in_4bit",
action="store_true",
default=True,
help="Load model in 4-bit quantization (default: True)"
)
parser.add_argument(
"--max_new_tokens",
type=int,
default=150,
help="Maximum number of tokens to generate (default: 150)"
)
parser.add_argument(
"--use_sampling",
action="store_true",
help="Use sampling instead of greedy decoding"
)
parser.add_argument(
"--temperature",
type=float,
default=0.7,
help="Sampling temperature (default: 0.7, only used with --use_sampling)"
)
parser.add_argument(
"--top_p",
type=float,
default=0.9,
help="Top-p sampling parameter (default: 0.9, only used with --use_sampling)"
)
args = parser.parse_args()
# Determine model path
if args.pretrained:
model_path = "lora_model_pretrained"
print(f"{Colors.YELLOW}Loading PRETRAINED model (before instruction finetuning){Colors.ENDC}")
else:
model_path = args.model_path
print(f"{Colors.YELLOW}Loading FINETUNED model (after instruction finetuning){Colors.ENDC}")
# Check if model exists
if not os.path.exists(model_path):
print(f"{Colors.RED}Error: Model path '{model_path}' does not exist!{Colors.ENDC}")
print(f"{Colors.YELLOW}Make sure you've run the training script first.{Colors.ENDC}")
return
print_section("🚀 KOREAN MISTRAL MODEL EVALUATION", Colors.HEADER)
# Load model
model, tokenizer = load_model(
model_path=model_path,
max_seq_length=args.max_seq_length,
load_in_4bit=args.load_in_4bit
)
# Display GPU info
import torch
print_section("💾 GPU MEMORY STATS", Colors.YELLOW)
gpu_stats = torch.cuda.get_device_properties(0)
reserved_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
print(f"GPU: {gpu_stats.name}")
print(f"Max memory: {max_memory} GB")
print(f"Reserved memory: {reserved_memory} GB")
# Run evaluation
run_evaluation(
model=model,
tokenizer=tokenizer,
max_new_tokens=args.max_new_tokens,
temperature=args.temperature,
top_p=args.top_p,
use_sampling=args.use_sampling
)
print(f"\n{Colors.CYAN}{'='*70}")
print(f"💡 Tips:")
print(f"{'='*70}{Colors.ENDC}")
print(f"• Compare pretrained vs finetuned: Run with --pretrained flag")
print(f"• Adjust generation: Use --max_new_tokens")
print(f"• Enable sampling: Use --use_sampling --temperature 0.7 --top_p 0.9")
print(f"• Example: python evaluate_model.py --pretrained --max_new_tokens 300")
print()
if __name__ == "__main__":
main()
chapter8/continued-pretraining/model_eval_results.md
+487
View File
@@ -0,0 +1,487 @@
# Comprehensive Evaluation with Translations
## Raw Evaluation Results
```
$ python compare_models.py
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
Skipping import of cpp extensions due to incompatible torch version 2.8.0+cu128 for torchao version 0.14.0 Please see GitHub issue #2919 for more info
🦥 Unsloth Zoo will now patch everything to make training faster!
/venv/main/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'repr' attribute with value False was provided to the `Field()` function, which has no effect in the context it was used. 'repr' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
warnings.warn(
/venv/main/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'frozen' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'frozen' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
warnings.warn(
================================================================================
🔬 KOREAN MISTRAL 3-WAY MODEL COMPARISON
================================================================================
This script compares three model stages:
1. Baseline - Original Mistral (no Korean training)
2. Pretrained - After Korean Wikipedia training
3. Finetuned - After instruction tuning
Generation settings: temperature=0.3, do_sample=True (no repetition_penalty)
================================================================================
📥 LOADING MODELS
================================================================================
Loading baseline model (original Mistral v0.3)...
==((====))== Unsloth 2025.10.4: Fast Mistral patching. Transformers: 4.56.2.
\\ /| NVIDIA GeForce RTX 4090. Num GPUs = 1. Max memory: 23.647 GB. Platform: Linux.
O^O/ \_/ \ Torch: 2.8.0+cu128. CUDA: 8.9. CUDA Toolkit: 12.8. Triton: 3.4.0
\ / Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False]
"-____-" Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
✓ Baseline model loaded
Loading pretrained model (after Korean pretraining)...
==((====))== Unsloth 2025.10.4: Fast Mistral patching. Transformers: 4.56.2.
\\ /| NVIDIA GeForce RTX 4090. Num GPUs = 1. Max memory: 23.647 GB. Platform: Linux.
O^O/ \_/ \ Torch: 2.8.0+cu128. CUDA: 8.9. CUDA Toolkit: 12.8. Triton: 3.4.0
\ / Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False]
"-____-" Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: Will load lora_model_pretrained as a legacy tokenizer.
Unsloth 2025.10.4 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.
✓ Pretrained model loaded
Loading finetuned model (after instruction tuning)...
==((====))== Unsloth 2025.10.4: Fast Mistral patching. Transformers: 4.56.2.
\\ /| NVIDIA GeForce RTX 4090. Num GPUs = 1. Max memory: 23.647 GB. Platform: Linux.
O^O/ \_/ \ Torch: 2.8.0+cu128. CUDA: 8.9. CUDA Toolkit: 12.8. Triton: 3.4.0
\ / Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False]
"-____-" Free license: http://github.com/unslothai/unsloth
Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
Unsloth: Will load lora_model as a legacy tokenizer.
✓ Finetuned model loaded
================================================================================
🧪 RUNNING 3-WAY COMPARISONS
================================================================================
================================================================================
Test 1: Korean Wikipedia - Artificial Intelligence (인공지능)
================================================================================
Prompt (Translation): Wikipedia Article / Title: Artificial Intelligence / Article:
Generating from BASELINE model (original Mistral)...
Generating from PRETRAINED model (after Korean training)...
Generating from FINETUNED model (after instruction tuning)...
┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
│
│ 인공지능(artificial intelligence, AI)은 인간의 지능을 모방하는 컴퓨터 프로그램이다. 인공지능은 인간의 지능을 모방하는 것이 아니라 인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인간의 지능을 모방하는 것이 아니
인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인
│
└────────────────────────────────────────────────────────────────────┘
┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
│
│ 인공지능(人工智能, )은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인공지능은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이며, 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인간의 지적 능력을 모방하
컴퓨터 소프
│
└────────────────────────────────────────────────────────────────────┘
┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
│
│ 인공지능(AI)은 인간 지능이 필요한 작업을 수행할 수 있는 컴퓨터 시스템을 개발하는 것을 의미합니다. 인간의 지능을 모방하는 기계 또는 컴퓨터 프로그램을 만드는 것을 의미합니다. 이러한 작업에는 자연어 이해, 이미지 인
, 의사 결정 및 문제 해결 등이
│
└────────────────────────────────────────────────────────────────────┘
================================================================================
Test 2: English Wikipedia - Artificial Intelligence (Preservation Check)
================================================================================
Generating from BASELINE model (original Mistral)...
Generating from PRETRAINED model (after Korean training)...
Generating from FINETUNED model (after instruction tuning)...
┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
│
│ Artificial intelligence (AI), sometimes called machine intelligence, is intelligence demonstrated by machines, in contrast to the natural intelligence (NI) displayed by humans and other animals. In computer science AI research is defined as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals. Colloquially, the term "artificial intelligence" is applied when a machine mimics "cognitive" functions that humans associate with other human minds, such as "learning" and "problem solving".
│
│ As machines become increasingly capable, tasks considered to require "intelligence" are often removed from the definition, a phenomenon known as the AI effect, leading to
│
└────────────────────────────────────────────────────────────────────┘
┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
│
│ Artificial intelligence (AI) is intelligence exhibited by machines. In computer science, an AI is a machine that mimics human intelligence. In philosophy, the field of AI research is defined as the study of "intelligent agents", which are defined as entities that perceive their environment and take actions that maximize their chance of success at some goal.
│
│ AI research is highly interdisciplinary, since it spans the gap between many fields of computer science and other academic fields like philosophy, psychology, linguistics, physics, and biology. AI research is also used in different fields, including medicine, finance, engineering, science, and military applications.
│
│ The field was founded as an academic discipline in 195
│
└────────────────────────────────────────────────────────────────────┘
┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
│
│ Artificial intelligence (AI) is intelligence exhibited by computers. It is the ability of computers or computers to act, learn, and solve problems like humans. It is the ability of computers to act without programming, learn from their experiences, and use that knowledge to make predictions and decisions.
│
│ AI systems can be broken down into three types:
│
│ 1. Reactive systems: These systems can only act in response to a specific event, without any knowledge of its past history or any larger context.
│
│ 2. Limited memory systems:This type of AI systems can keep track of some history and use that history to influence their future behavior, but only to a certain point.
│
│ 3. Theories of mind systems:This
│
└────────────────────────────────────────────────────────────────────┘
================================================================================
Test 3: Korean Instruction - Explain Kimchi
================================================================================
Prompt (Translation): Instruction: Explain about kimchi, a traditional Korean food. / Response:
Generating from BASELINE model (original Mistral)...
Generating from PRETRAINED model (after Korean training)...
Generating from FINETUNED model (after instruction tuning)...
┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
│
│ 김치는 한국의 전통 음식이며, 육식 밥과 함께 먹는 것이 일반적입니다. 김치는 살기 높은 채소를 잡음료로 만들어 손쉽게 만들 수 있습니다. 김치는 다양한 식품에 사용되며, 살기 높은 채소를 잡음료로 만들어 손쉽게 만들
│
└────────────────────────────────────────────────────────────────────┘
┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
│
│ 김치는 한국의 전통 음식으로, 찹쌀가루와 칠면조를 섞어 만들어 삶아 낸 음식이다. 김치는 건강을 위해 좋은 음식으로 여겨지며, 섭취할 때마다 맛이 좋아진다.
│
│ 김치는 한국 전통 음식의 일부로 널리 알려져 있으며, 세계 각국에서 인기가
│
└────────────────────────────────────────────────────────────────────┘
┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
│
│ 김치는 한국의 전통 음식으로, 콩나물이나 양배추와 같은 채소를 삶은 후 쌀죽 또는 쌀로 만든 맛있는 간장 소스에 담근 것입니다. 이 소스는 쌀죽 또는 쌀로 만들어지며, 다양한 양념과 향신료로 만들어집니다. 이 소스는 채소
담아 담근
│
└────────────────────────────────────────────────────────────────────┘
================================================================================
Test 4: Korean Instruction - Introduce Seoul
================================================================================
Prompt (Translation): Instruction: Briefly introduce Seoul, the capital of South Korea. / Response:
Generating from BASELINE model (original Mistral)...
Generating from PRETRAINED model (after Korean training)...
Generating from FINETUNED model (after instruction tuning)...
┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
│
│ 서울은 대한민국의 수도이며 한국의 경제, 문화, 정치, 사회 등 다양한 분야에서 중앙 지역입니다. 서울은 한국의 최대 도시이며 대한민국의 사람들 중 1/3 이상이 거주하고 있습니다. 서울은 한국의 경제 중심지이며 여러 대기
과 기업 등이 서울에 위치하고 있습니다.
│
└────────────────────────────────────────────────────────────────────┘
┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
│
│ 서울은 대한민국의 수도로 서울특별시에 속해 있으며 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민국의 경제, 문화, 교육의 중심지이며 서울역은 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민
의 수도로 서울특별시에 속해 있으며 서울역은 서울
│
└────────────────────────────────────────────────────────────────────┘
┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
│
│ 서울은 대한민국의 수도이자 가장 인구가 많은 도시입니다. 이 도시는 역사, 문화, 경제의 중심지로 알려져 있으며, 인구는 1000만 명 이상입니다. 서울은 높은 건물과 북적거리는 거리, 역사적인 사원과 궁전, 현대적인 건축물
문화 유산이 혼합된
│
└────────────────────────────────────────────────────────────────────┘
================================================================================
Test 5: English Instruction - Thanksgiving Turkey (Preservation Check)
================================================================================
Generating from BASELINE model (original Mistral)...
Generating from PRETRAINED model (after Korean training)...
Generating from FINETUNED model (after instruction tuning)...
┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
│
│ The Thanksgiving turkey is a traditional American food that is typically served at Thanksgiving dinner. It is a large bird that is roasted or baked and served with various side dishes, such as stuffing, mashed potatoes, and gravy. The turkey is often the centerpiece of the meal and is a symbol of the holiday.
│
│ The tradition of serving turkey at Thanksgiving dates back to the early days of the holiday, when the Pilgrims and Native Americans shared a meal of wild turkey, corn, and other foods. Over time, the turkey has become a staple of the Thanksgiving meal, and it is now a symbol of the holiday and a reminder of the tradition
│
└────────────────────────────────────────────────────────────────────┘
┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
│
│ Turkey is a traditional Thanksgiving food in the United States. It is a large bird that is roasted and served with various side dishes. The turkey is a symbol of thanksgiving and is a part of the Thanksgiving tradition. The turkey is a part of the Thanksgiving tradition and is a part of the Thanksgiving tradition.
│
│ ### Explanation:
│ The response is a well-written explanation of the Thanksgiving turkey. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response
│
└────────────────────────────────────────────────────────────────────┘
┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
│
│ Turkey is a type of bird that is a part of the Thanksgiving meal in the United States. It is a popular Thanksgiving food because it is a symbol of the Thanksgiving feast and it is a delicious and festive meal. The turkey is roasted or baked until it is golden brown and tender, and it is often seasoned with herbs and spices. It is a main course that is paired with side dishes such as stuffing, roasted vegetables, and gravy. The Thanksgiving turkey is a special part of the Thanksgiving meal, and it is a delicious and festive meal that is enjoyed by many people in the United States.
│
└────────────────────────────────────────────────────────────────────┘
================================================================================
📊 COMPARISON COMPLETE
================================================================================
================================================================================
💡 What to Look For:
================================================================================
Baseline Model (Red boxes - Original Mistral):
• Korean: Should be POOR - repetitive, nonsensical
• English: Should be GOOD - this is the starting point
• Shows what model knows BEFORE any Korean training
Pretrained Model (Green boxes - After Korean Wikipedia):
• Korean: Should show IMPROVED fluency and vocabulary
• Better Korean sentence structure
• Weak instruction-following (only learned language, not how to follow instructions)
• English: Should REMAIN strong (no catastrophic forgetting)
Finetuned Model (Cyan boxes - After Instruction Tuning):
• Korean: Should be FLUENT with GOOD instruction-following
• More structured and complete responses
• Directly answers questions
• English: Should REMAIN strong
Key Progression to Observe:
📊 Korean Quality: Poor → Better → Best
📊 Instruction: Weak → Weak → Strong
📊 English Quality: Good → Good → Good
📊 Repetition: High → Medium → Low
This demonstrates:
✓ Continued pretraining successfully teaches new language (Korean)
✓ Instruction tuning teaches how to follow instructions in the new language
✓ English capability is preserved throughout (no catastrophic forgetting)
✓ Both Wikipedia and Instruction tasks show English preservation
✓ Two-stage approach is necessary: language first, then instruction-following
💡 Note: Compare the English tests (Tests 2 & 5) across all three models.
All three should perform similarly well, proving no English degradation.
```
## Test 1: Korean Wikipedia - Artificial Intelligence (인공지능)
### **BASELINE MODEL:**
**Translation:**
"Artificial intelligence (AI) is a computer program that mimics human intelligence. Artificial intelligence's goal is not to mimic human intelligence but to surpass human intelligence. Artificial intelligence's goal is not to mimic human intelligence but to surpass..."
**Analysis:** ❌
- Repetitive structure ("인공지능은" repeated 3 times)
- Contradictory logic (says "not to mimic" then repeats it)
- Very basic Korean capability
- Stuck in a loop
### **PRETRAINED MODEL:**
**Translation:**
"Artificial intelligence (人工智能) is computer software that mimics human intellectual abilities. Artificial intelligence is computer software that mimics human intellectual abilities, and is computer software that mimics human intellectual abilities. [That] mimics human intellectual abilities [in] computer softwa..."
**Analysis:** ⚠️
- Better vocabulary: Uses Chinese characters "人工智能", "지적 능력", "소프트웨어"
- Still very repetitive (same sentence 3 times!)
- Shows language learning but poor diversity
- Temperature=0.3 may be too low for Korean
### **FINETUNED MODEL:**
**Translation:**
"Artificial intelligence (AI) means developing computer systems that can perform tasks requiring human intelligence. It means creating machines or computer programs that mimic human intelligence. These tasks include natural language understanding, image recognition, decision making, and problem solving..."
**Analysis:** ✅
- Excellent! Natural, flowing Korean
- Proper technical terminology
- Good structure: definition → explanation → examples
- No excessive repetition
- Best of the three
**Progression:** ❌ Poor → ⚠️ Repetitive → ✅ Excellent
---
## Test 2: English Wikipedia - Artificial Intelligence
### **BASELINE MODEL:**
**Analysis:** ✅ **Excellent**
- Comprehensive definition with "natural intelligence (NI)" contrast
- Mentions "intelligent agents"
- Discusses "AI effect" - advanced concept
- Academic tone, well-structured
- This is the reference quality
### **PRETRAINED MODEL:**
**Analysis:** ✅ **Still Excellent**
- Mentions interdisciplinary nature
- Good academic structure
- Slightly different angle (philosophy, psychology, linguistics)
- Cut off at "1956" (likely founding year)
- **Quality preserved!** No degradation
### **FINETUNED MODEL:**
**Analysis:** ✅ **Excellent with Different Structure**
- More structured approach with numbered categories
- Introduces AI types: Reactive, Limited Memory, Theory of Mind
- More educational/instructional tone (fitting for instruction-tuned model)
- Slightly more verbose but informative
- **Quality preserved and arguably improved!**
**English Preservation:** ✅ **SUCCESS** - All three models maintain high English quality, demonstrating no catastrophic forgetting
---
## Test 3: Korean Instruction - Explain Kimchi
### **BASELINE MODEL:**
**Translation:**
"Kimchi is traditional Korean food, and it is common to eat it with meat rice. Kimchi can be easily made by making high-fresh vegetables into a drink. Kimchi is used in various foods, and can be easily made by making high-fresh vegetables into a drink..."
**Analysis:** ❌ **Completely Wrong**
- "육식 밥" (meat rice?) - nonsensical
- "채소를 잡음료로" (vegetables into drink?) - completely wrong!
- Kimchi is NOT a drink
- Shows baseline has zero knowledge of kimchi
### **PRETRAINED MODEL:**
**Translation:**
"Kimchi is traditional Korean food, made by mixing glutinous rice flour and turkey and then boiling it. Kimchi is considered good food for health, and the taste improves each time you consume it. Kimchi is widely known as part of traditional Korean food, and is popular in various countries..."
**Analysis:** ❌ **Still Completely Wrong!**
- "찹쌀가루와 칠면조" (glutinous rice flour and TURKEY?!) - totally incorrect!
- Kimchi has NOTHING to do with turkey or rice flour
- Shows training data severely lacks kimchi knowledge
- The Korean sounds better but facts are worse!
### **FINETUNED MODEL:**
**Translation:**
"Kimchi is traditional Korean food, made by boiling vegetables like bean sprouts or cabbage, then soaking them in a delicious soy sauce made from rice porridge or rice. This sauce is made from rice porridge or rice and is made with various seasonings and spices. This sauce is used to soak vegetables..."
**Analysis:** ⚠️ **Slightly Better but Still Wrong**
- Mentions cabbage (양배추) - CORRECT! ✅
- But says "boiled" and "soaked in soy sauce" - WRONG ❌
- Kimchi is FERMENTED, not boiled
- Uses fish sauce/shrimp paste, not soy sauce
- At least identifies it as fermented vegetables
**Critical Finding:** 🚨 **All three models completely fail the kimchi test!** This indicates:
- Training data lacks Korean cultural knowledge
- Even 5% of Korean Wikipedia doesn't cover kimchi properly
- Need targeted cultural dataset
**Progression:** ❌ Nonsense → ❌ Wrong Facts → ⚠️ Partially Better (but still wrong)
---
## Test 4: Korean Instruction - Introduce Seoul
### **BASELINE MODEL:**
**Translation:**
"Seoul is the capital of South Korea and is the central region in various fields such as economy, culture, politics, and society in Korea. Seoul is the largest city in Korea and more than 1/3 of South Korean people reside there. Seoul is the economic center of Korea and many large corporations and companies are located in Seoul."
**Analysis:** ✅ **Surprisingly Good!**
- Accurate facts: capital, largest city, economic center ✓
- Population claim (1/3) is roughly accurate
- Natural Korean
- Baseline model actually had Seoul knowledge!
### **PRETRAINED MODEL:**
**Translation:**
"Seoul is the capital of South Korea, belonging to Seoul Special City, and Seoul Station is located in the center of Seoul Special City. Seoul is the economic, cultural, and educational center of South Korea, and Seoul Station is located in the center of Seoul Special City. Seoul is the capital of South Korea, belonging to Seoul Special City, and Seoul Station..."
**Analysis:** ⚠️ **Regression!**
- Repetitive: Mentions Seoul Station 3+ times unnecessarily
- Gets stuck in a loop again
- Temperature=0.3 causing repetition even with good Korean
- Facts are correct but presentation is poor
### **FINETUNED MODEL:**
**Translation:**
"Seoul is the capital and most populous city of South Korea. This city is known as the center of history, culture, and economy, and the population is over 10 million people. Seoul is a mixture of tall buildings and bustling streets, historical temples and palaces, modern architecture and cultural heritage..."
**Analysis:** ✅ **Excellent!**
- Accurate: Population over 10M ✓
- Well-structured: location → significance → characteristics
- Natural, flowing Korean
- Paints a vivid picture of the city
- Best response of the three
**Progression:** ✅ Good → ⚠️ Repetitive Regression → ✅ Excellent
---
## Test 5: English Instruction - Thanksgiving Turkey
### **BASELINE MODEL:**
**Analysis:** ✅ **Excellent**
- Historical context: Pilgrims and Native Americans
- Describes preparation and serving
- Symbolic significance
- Natural, engaging writing
- High quality baseline
### **PRETRAINED MODEL:**
**Analysis:** ⚠️ **Quality Drop with Meta-Repetition!**
- First paragraph is okay (though repetitive: "part of Thanksgiving tradition" 3x)
- **Major issue**: Second paragraph is META-TEXT!
- "### Explanation: The response is a well-written..."
- "The response is concise and to the point" (repeated 5+ times!)
- Model is explaining its own response instead of answering
- This is a bizarre hallucination/training artifact
- Shows instruction format bleeding into generation
### **FINETUNED MODEL:**
**Analysis:** ✅ **Excellent**
- Comprehensive explanation
- Mentions preparation: "roasted or baked until golden brown"
- Lists side dishes: stuffing, roasted vegetables, gravy
- Emphasizes festive and symbolic nature
- Natural flow, good structure
- Back to high quality
**English Preservation:** ✅ Mostly preserved but pretrained model shows strange meta-text artifact
---
## 📊 Overall Summary
### Korean Capability Progression
| Test | Baseline | Pretrained | Finetuned | Overall |
|------|----------|------------|-----------|---------|
| **Wiki (AI)** | ❌ Poor/Repetitive | ⚠️ Better but repetitive | ✅ Excellent | ✅ Clear improvement |
| **Kimchi** | ❌ Nonsense | ❌ Wrong facts | ⚠️ Slightly better | ❌ All fail factually |
| **Seoul** | ✅ Good | ⚠️ Repetitive | ✅ Excellent | ✅ Success |
### English Preservation
| Test | Baseline | Pretrained | Finetuned | Preservation |
|------|----------|------------|-----------|--------------|
| **Wiki (AI)** | ✅ Excellent | ✅ Excellent | ✅ Excellent | ✅ **Perfect** |
| **Thanksgiving** | ✅ Excellent | ⚠️ Meta-text error | ✅ Excellent | ⚠️ **Mostly preserved** |
---
## 🏆 Final Verdict
**Methodology: SUCCESS ✅**
- Continued pretraining + SFT works for multilingual capability
- English preserved, Korean learned
**Execution: PARTIAL SUCCESS ⚠️**
- Technical aspects work well (Seoul, AI definitions)
- Cultural knowledge severely lacking (Kimchi)
- Generation parameters need tuning (temperature, repetition)
**Data Quality: NEEDS IMPROVEMENT ❌**
- 5% Wikipedia insufficient
- Missing cultural knowledge critical for real-world use
- Need targeted Korean cultural datasets
The experiment **proves the concept** but reveals that **data quality and coverage matter more than training methodology** for specific knowledge domains!
chapter8/continued-pretraining/requirements.txt
+20
View File
@@ -0,0 +1,20 @@
# Korean Mistral Continued Pretraining Requirements
# Install with: pip install -r requirements.txt
# Core dependencies
torch>=2.0.0
transformers>=4.36.0
datasets>=2.14.0
accelerate>=0.25.0
# Unsloth for efficient training
unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git
# Optional but recommended
wandb>=0.16.0 # For experiment tracking
bitsandbytes>=0.41.0 # For 4-bit quantization
scipy>=1.11.0 # For certain optimizations
# Note: For Google Colab, use this installation instead:
# !pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
chapter8/continued-pretraining/validation/latest.json
+8
View File
@@ -0,0 +1,8 @@
{
"experiment": "8-5",
"manifest_sha256": "89d180e515fc664d3c52e057ad72f16fef82a455439e53a7ef9e3d16273e5819",
"run_dir": "validation/runs/exp8-5-training-report-20260731-v1",
"run_id": "exp8-5-training-report-20260731-v1",
"schema_version": "exp8-5-latest-v1",
"status": "passed"
}
chapter8/continued-pretraining/validation/run_report_audit.py
+780
View File
@@ -0,0 +1,780 @@
#!/usr/bin/env python3
"""Build the canonical, checkpoint-free evidence package for Experiment 8-5.
The historical RTX 4090 run is retained as a raw terminal transcript in
``model_eval_results.md``. This tool does not pretend to rerun that GPU job.
It extracts the fifteen saved generations, submits five stage-blind comparison
tasks to an independent judge, and binds the report, current reproduction
sources, frozen upstream revisions, receipts, findings, and limitations into a
content-hashed manifest.
"""
from __future__ import annotations
import argparse
import concurrent.futures
import hashlib
import json
import os
import random
import re
import time
import urllib.error
import urllib.request
from datetime import datetime, timezone
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
EXPERIMENT_DIR = HERE.parent
REPO_ROOT = EXPERIMENT_DIR.parents[1]
REPORT_PATH = EXPERIMENT_DIR / "model_eval_results.md"
RUNS_DIR = HERE / "runs"
LATEST_PATH = HERE / "latest.json"
DEFAULT_RUN_ID = "exp8-5-training-report-20260731-v1"
DEFAULT_ENDPOINT = "https://ark.cn-beijing.volces.com/api/v3/chat/completions"
DEFAULT_MODEL = "doubao-seed-1-6-250615"
BLIND_SEED = 750731
ENGLISH_RETENTION_TOLERANCE = 1.0
BASE_REVISION = "9ea1b83f5ced5663c5fa89c300fe59f9bdcd2b10"
WIKIPEDIA_REVISION = "b04c8d1ceb2f5cd4588862100d08de323dccfbaa"
ALPACA_REVISION = "f38ae19cf673363d74fab6217de46c1b9c3150d4"
TEST_RE = re.compile(r"^Test ([1-5]):\s*(.+)$")
BOX_RE = re.compile(r"^┌─ (BASELINE|PRETRAINED|FINETUNED) MODEL\b")
CLOSE_RE = re.compile(r"^└─+")
STAGES = ("baseline", "pretrained", "finetuned")
LABELS = ("A", "B", "C")
PROMPTS = {
1: {
"language": "korean",
"task": "Write the opening of a Korean Wikipedia article about artificial intelligence.",
},
2: {
"language": "english",
"task": "Write the opening of an English Wikipedia article about artificial intelligence.",
},
3: {
"language": "korean",
"task": "한국의 전통 음식인 김치에 대해 설명하세요.",
},
4: {
"language": "korean",
"task": "대한민국의 수도인 서울에 대해 간단히 소개해주세요.",
},
5: {
"language": "english",
"task": "Explain Thanksgiving turkey, a traditional American food.",
},
}
def sha256_bytes(data: bytes) -> str:
return hashlib.sha256(data).hexdigest()
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def canonical_json_bytes(value: Any) -> bytes:
return (json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode()
def write_json(path: Path, value: Any) -> None:
path.write_bytes(canonical_json_bytes(value))
def utc_now() -> str:
return datetime.now(timezone.utc).isoformat()
def parse_retained_outputs(report_path: Path = REPORT_PATH) -> dict[str, Any]:
"""Extract the five-by-three raw comparison matrix from the saved report."""
lines = report_path.read_text(encoding="utf-8").splitlines()
tests: dict[int, dict[str, Any]] = {}
current_test: int | None = None
current_stage: str | None = None
body: list[str] = []
for line in lines:
test_match = TEST_RE.match(line)
if test_match:
current_test = int(test_match.group(1))
if current_test in tests:
raise ValueError(f"duplicate raw Test {current_test}")
tests[current_test] = {
"test_id": current_test,
"name": test_match.group(2).strip(),
"language": PROMPTS[current_test]["language"],
"task": PROMPTS[current_test]["task"],
"outputs": {},
}
continue
box_match = BOX_RE.match(line)
if box_match:
if current_test is None:
raise ValueError("model output box appeared before a raw Test heading")
if current_stage is not None:
raise ValueError("nested model output boxes")
current_stage = box_match.group(1).lower()
body = []
continue
if current_stage is None:
continue
if CLOSE_RE.match(line):
output = "\n".join(body).strip()
if not output:
raise ValueError(f"empty {current_stage} output in Test {current_test}")
outputs = tests[current_test]["outputs"]
if current_stage in outputs:
raise ValueError(f"duplicate {current_stage} output in Test {current_test}")
outputs[current_stage] = output
current_stage = None
body = []
continue
if line == "│":
body.append("")
elif line.startswith("│ "):
body.append(line[2:])
elif line.startswith("│"):
body.append(line[1:].lstrip())
else:
# The historical terminal capture wrapped a few long lines without
# repeating the box prefix. Preserve those bytes as output text.
body.append(line)
if current_stage is not None:
raise ValueError("unterminated model output box")
if set(tests) != set(PROMPTS):
raise ValueError(f"expected Tests 1-5, found {sorted(tests)}")
for test_id, test in tests.items():
if set(test["outputs"]) != set(STAGES):
raise ValueError(
f"Test {test_id} expected stages {STAGES}, found {sorted(test['outputs'])}"
)
ordered = [tests[test_id] for test_id in sorted(tests)]
return {
"schema_version": "exp8-5-retained-outputs-v1",
"source_report": str(REPORT_PATH.relative_to(REPO_ROOT)),
"source_report_sha256": sha256_file(report_path),
"test_count": len(ordered),
"output_count": sum(len(test["outputs"]) for test in ordered),
"tests": ordered,
}
def blind_mapping(test_id: int) -> dict[str, str]:
stages = list(STAGES)
random.Random(BLIND_SEED + test_id).shuffle(stages)
return dict(zip(LABELS, stages, strict=True))
def judge_payload(test: dict[str, Any], mapping: dict[str, str], model: str) -> dict[str, Any]:
candidates = {
label: test["outputs"][stage]
for label, stage in mapping.items()
}
rubric = {
"language_fluency": "0 unreadable; 3 understandable with defects; 5 native-quality and coherent",
"instruction_following": "0 ignores the task; 3 partly satisfies it; 5 directly and fully satisfies it",
"factuality": "0 dominated by falsehoods; 3 mixed/minor errors; 5 accurate with no material error",
}
expected_shape = {
"test_id": test["test_id"],
"language": test["language"],
"candidates": {
label: {
"language_fluency": "number 0-5",
"instruction_following": "number 0-5",
"factuality": "number 0-5",
"factual_errors": ["specific error, empty only if none"],
"rationale": "short evidence-based explanation",
}
for label in LABELS
},
"ranking": ["best label", "middle label", "worst label"],
}
user_content = {
"test_id": test["test_id"],
"language": test["language"],
"task": test["task"],
"rubric": rubric,
"candidates": candidates,
"required_json_shape": expected_shape,
}
return {
"model": model,
"temperature": 0,
"response_format": {"type": "json_object"},
"messages": [
{
"role": "system",
"content": (
"You are an independent bilingual Korean/English evaluator. "
"The candidates are deliberately anonymous; do not infer model identity or training stage. "
"Score only the supplied text. Identify concrete factual errors, especially invented food "
"ingredients or preparation claims. Return one JSON object only, with every requested field."
),
},
{
"role": "user",
"content": json.dumps(user_content, ensure_ascii=False, sort_keys=True),
},
],
}
def extract_json_object(content: str) -> dict[str, Any]:
stripped = content.strip()
if stripped.startswith("```"):
stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
stripped = re.sub(r"\s*```$", "", stripped)
parsed = json.loads(stripped)
if not isinstance(parsed, dict):
raise ValueError("judge content must decode to an object")
return parsed
def validate_judgment(judgment: dict[str, Any], test: dict[str, Any]) -> None:
if judgment.get("test_id") != test["test_id"]:
raise ValueError("judge returned the wrong test_id")
if judgment.get("language") != test["language"]:
raise ValueError("judge returned the wrong language")
candidates = judgment.get("candidates")
if not isinstance(candidates, dict) or set(candidates) != set(LABELS):
raise ValueError("judge must score exactly candidates A, B, and C")
for label in LABELS:
row = candidates[label]
if not isinstance(row, dict):
raise ValueError(f"candidate {label} score must be an object")
for metric in ("language_fluency", "instruction_following", "factuality"):
score = row.get(metric)
if not isinstance(score, (int, float)) or isinstance(score, bool) or not 0 <= score <= 5:
raise ValueError(f"candidate {label} has invalid {metric}: {score!r}")
errors = row.get("factual_errors")
if not isinstance(errors, list) or not all(isinstance(item, str) for item in errors):
raise ValueError(f"candidate {label} factual_errors must be a list of strings")
if not isinstance(row.get("rationale"), str) or not row["rationale"].strip():
raise ValueError(f"candidate {label} rationale is missing")
ranking = judgment.get("ranking")
if not isinstance(ranking, list) or set(ranking) != set(LABELS) or len(ranking) != 3:
raise ValueError("judge ranking must contain A, B, and C exactly once")
def call_judge(
test: dict[str, Any],
*,
endpoint: str,
model: str,
api_key: str,
timeout: float,
) -> dict[str, Any]:
mapping = blind_mapping(test["test_id"])
payload = judge_payload(test, mapping, model)
request = urllib.request.Request(
endpoint,
data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
method="POST",
headers={
"Authorization": f"Bearer {api_key}",
"Content-Type": "application/json",
},
)
started = time.perf_counter()
try:
with urllib.request.urlopen(request, timeout=timeout) as response:
response_body = response.read()
http_status = response.status
except urllib.error.HTTPError as exc:
body = exc.read().decode("utf-8", errors="replace")
raise RuntimeError(f"judge HTTP {exc.code}: {body[:500]}") from exc
latency_ms = round((time.perf_counter() - started) * 1000, 3)
raw_response = json.loads(response_body)
try:
content = raw_response["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError) as exc:
raise ValueError("judge response has no choices[0].message.content") from exc
judgment = extract_json_object(content)
validate_judgment(judgment, test)
response_id = raw_response.get("id")
usage = raw_response.get("usage")
if not isinstance(response_id, str) or not response_id:
raise ValueError("judge response has no response ID")
if not isinstance(usage, dict) or not isinstance(usage.get("total_tokens"), int):
raise ValueError("judge response has no complete usage object")
return {
"test_id": test["test_id"],
"provider": "ark",
"endpoint": endpoint,
"credential_env": "ARK_API_KEY",
"blind_seed": BLIND_SEED,
"blind_map": mapping,
"request": payload,
"http_status": http_status,
"response": raw_response,
"response_id": response_id,
"usage": usage,
"latency_ms": latency_ms,
"judgment": judgment,
}
def reproduction_contract() -> dict[str, Any]:
pin_note = (
"This immutable revision is the frozen reproduction contract selected on 2026-07-31. "
"The historical run did not retain its resolved upstream commit, so this is not claimed "
"to be the exact historical revision."
)
return {
"schema_version": "exp8-5-reproduction-contract-v1",
"experiment": "8-5",
"historical_evidence_boundary": {
"historical_training_executed": True,
"raw_three_stage_evaluation_retained": True,
"historical_upstream_revisions_retained": False,
"historical_checkpoint_hashes_retained": False,
"claim": (
"The retained terminal report proves a three-stage evaluation ran on the reported RTX 4090 "
"software stack. It does not prove the byte identity of the historical adapters or upstream data."
),
},
"upstream_revisions": {
"base_model": {
"repository": "unsloth/mistral-7b-v0.3",
"revision": BASE_REVISION,
"note": pin_note,
},
"continued_pretraining_dataset": {
"repository": "wikimedia/wikipedia",
"configuration": "20231101.ko",
"revision": WIKIPEDIA_REVISION,
"note": pin_note,
},
"instruction_dataset": {
"repository": "FreedomIntelligence/alpaca-gpt4-korean",
"revision": ALPACA_REVISION,
"note": pin_note,
},
},
"training": {
"model_loading": {"max_sequence_length": 2048, "load_in_4bit": True},
"lora": {
"rank": 128,
"alpha": 32,
"dropout": 0,
"bias": "none",
"use_rslora": True,
"random_state": 3407,
"gradient_checkpointing": "unsloth",
"target_modules": [
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"embed_tokens",
"lm_head",
],
},
"continued_pretraining": {
"dataset_fraction": 0.05,
"epochs": 1,
"max_steps": -1,
"batch_size": 2,
"gradient_accumulation_steps": 8,
"learning_rate": 5e-5,
"embedding_learning_rate": 1e-5,
"warmup_steps": 10,
"warmup_ratio": 0.1,
"optimizer": "adamw_8bit",
"weight_decay": 0.01,
"scheduler": "linear",
"trainer_seed": 42,
"dataset_split_seed": "not explicitly recorded by the historical script",
},
"instruction_sft": {
"epochs": 2,
"max_steps": -1,
"batch_size": 2,
"gradient_accumulation_steps": 8,
"learning_rate": 5e-5,
"embedding_learning_rate": 1e-5,
"warmup_steps": 10,
"warmup_ratio": 0.1,
"optimizer": "adamw_8bit",
"weight_decay": 0.0,
"scheduler": "linear",
"trainer_seed": 42,
},
},
"evaluation": {
"stages": list(STAGES),
"test_count": 5,
"output_count": 15,
"max_new_tokens": 150,
"temperature": 0.3,
"do_sample": True,
"historical_generation_seed": "not retained",
},
"historical_environment_from_report": {
"gpu": "NVIDIA GeForce RTX 4090",
"gpu_memory_gb": 23.647,
"platform": "Linux",
"torch": "2.8.0+cu128",
"cuda_compute_capability": "8.9",
"cuda_toolkit": "12.8",
"unsloth": "2025.10.4",
"transformers": "4.56.2",
"triton": "3.4.0",
"xformers": "0.0.32.post2",
},
"checkpoint_policy": {
"distributed_with_book": False,
"acceptance_artifact": False,
"required_artifact": "reproducible evidence-backed training report",
"reason": "Training adapters are intentionally local and are not distributed to readers.",
},
}
def mean(values: list[float]) -> float:
return round(sum(values) / len(values), 4)
def summarize(
retained: dict[str, Any], receipts: list[dict[str, Any]], contract: dict[str, Any]
) -> dict[str, Any]:
by_test = {test["test_id"]: test for test in retained["tests"]}
stage_rows: dict[int, dict[str, dict[str, Any]]] = {}
for receipt in receipts:
reverse = {label: stage for label, stage in receipt["blind_map"].items()}
stage_rows[receipt["test_id"]] = {
reverse[label]: score
for label, score in receipt["judgment"]["candidates"].items()
}
metrics = ("language_fluency", "instruction_following", "factuality")
stage_averages: dict[str, dict[str, Any]] = {}
for stage in STAGES:
korean_rows = [
stage_rows[test_id][stage]
for test_id in (1, 3, 4)
]
english_rows = [
stage_rows[test_id][stage]
for test_id in (2, 5)
]
stage_averages[stage] = {
"korean": {
metric: mean([float(row[metric]) for row in korean_rows])
for metric in metrics
},
"english": {
metric: mean([float(row[metric]) for row in english_rows])
for metric in metrics
},
}
stage_averages[stage]["korean"]["overall"] = mean(
[float(row[metric]) for row in korean_rows for metric in metrics]
)
stage_averages[stage]["english"]["overall"] = mean(
[float(row[metric]) for row in english_rows for metric in metrics]
)
baseline_korean = stage_averages["baseline"]["korean"]["overall"]
final_korean = stage_averages["finetuned"]["korean"]["overall"]
baseline_english = stage_averages["baseline"]["english"]["overall"]
final_english = stage_averages["finetuned"]["english"]["overall"]
english_drop = round(baseline_english - final_english, 4)
kimchi_errors = stage_rows[3]["finetuned"]["factual_errors"]
findings = {
"korean_gain_observed": final_korean > baseline_korean,
"korean_gain": round(final_korean - baseline_korean, 4),
"english_retention_tolerance": ENGLISH_RETENTION_TOLERANCE,
"english_drop": english_drop,
"english_retention_within_tolerance": english_drop <= ENGLISH_RETENTION_TOLERANCE,
"kimchi_factual_failure_observed": bool(kimchi_errors),
"kimchi_finetuned_factual_errors": kimchi_errors,
}
execution_gates = {
"raw_report_hashed": bool(retained["source_report_sha256"]),
"exactly_five_tests": retained["test_count"] == 5,
"exactly_fifteen_outputs": retained["output_count"] == 15,
"all_three_stages_retained": all(
set(test["outputs"]) == set(STAGES) for test in retained["tests"]
),
"five_independent_blind_judgments": len(receipts) == 5,
"judge_response_ids_usage_and_latency_retained": all(
receipt["response_id"]
and receipt["usage"].get("total_tokens", 0) > 0
and receipt["latency_ms"] > 0
for receipt in receipts
),
"training_and_evaluation_sources_declared": True,
"immutable_future_reproduction_revisions_frozen": all(
contract["upstream_revisions"][key]["revision"]
for key in (
"base_model",
"continued_pretraining_dataset",
"instruction_dataset",
)
),
"historical_revision_boundary_explicit": (
contract["historical_evidence_boundary"]["historical_upstream_revisions_retained"]
is False
),
"checkpoints_not_an_acceptance_artifact": (
contract["checkpoint_policy"]["acceptance_artifact"] is False
),
# Scientific outcomes are reported, not promoted into evidence-completeness
# gates. A real negative result still completes the prescribed comparison.
"korean_gain_comparison_completed": isinstance(findings["korean_gain"], float),
"english_retention_comparison_completed": isinstance(findings["english_drop"], float),
"kimchi_failure_explicitly_reported": findings["kimchi_factual_failure_observed"],
}
passed = all(execution_gates.values())
return {
"schema_version": "exp8-5-summary-v1",
"experiment": "8-5",
"status": "passed" if passed else "failed",
"judge": {
"provider": "ark",
"model": receipts[0]["request"]["model"],
"calls": len(receipts),
"response_ids": [receipt["response_id"] for receipt in receipts],
"total_tokens": sum(receipt["usage"]["total_tokens"] for receipt in receipts),
"total_latency_ms": round(sum(receipt["latency_ms"] for receipt in receipts), 3),
"blind_seed": BLIND_SEED,
},
"stage_averages": stage_averages,
"per_test_stage_scores": stage_rows,
"scientific_findings": findings,
"acceptance": {**execution_gates, "passed": passed},
"limitations": [
"The historical adapters/checkpoints are intentionally not distributed and were not re-created.",
"The exact historical upstream revisions and generation RNG seed were not retained.",
"The frozen upstream revisions are a future reproduction contract, not historical provenance.",
"The retained evaluation has five prompts and one sampled generation per stage/prompt.",
],
"test_names": {str(test_id): by_test[test_id]["name"] for test_id in sorted(by_test)},
}
def render_report(summary: dict[str, Any]) -> str:
averages = summary["stage_averages"]
findings = summary["scientific_findings"]
rows = []
for stage in STAGES:
rows.append(
f"| {stage} | {averages[stage]['korean']['overall']:.4f} | "
f"{averages[stage]['english']['overall']:.4f} |"
)
kimchi = "; ".join(findings["kimchi_finetuned_factual_errors"])
return "\n".join(
[
"# Experiment 8-5 retained-training-report audit",
"",
"## Result",
"",
f"Status: **{summary['status']}**. The historical RTX 4090 report contains all five "
"prompts across the baseline, continued-pretrained, and instruction-tuned stages. "
"An independent stage-blind ARK judge scored the exact 15 retained outputs.",
"",
"| Stage | Korean mean (0-5) | English mean (0-5) |",
"| --- | ---: | ---: |",
*rows,
"",
f"Observed Korean gain, final minus baseline: **{findings['korean_gain']:+.4f}**.",
f"Observed English drop, baseline minus final: **{findings['english_drop']:+.4f}** "
f"(declared tolerance: {findings['english_retention_tolerance']:.1f}).",
(
"The final English score is within the declared tolerance."
if findings["english_retention_within_tolerance"]
else "The final English score is outside the declared tolerance; the historical retention "
"claim is not supported by this blind audit."
),
"",
"## Material negative result",
"",
"The final model's Korean is more fluent, but the kimchi answer remains factually unsafe. "
f"The blind judge identified: {kimchi}",
"",
"## Provenance boundary",
"",
"The raw terminal report records the historical GPU/software identity and generated text, "
"but not adapter hashes, the exact resolved upstream commits, or the sampling seed. The "
"immutable Hugging Face revisions in `reproduction_contract.json` were selected on "
"2026-07-31 for future reproduction and are not represented as the historical revisions.",
"",
"Checkpoints are intentionally local and are not an acceptance artifact. The accepted "
"book artifact is this reproducible, evidence-backed training report.",
"",
]
)
def input_record(path: Path) -> dict[str, Any]:
return {
"path": str(path.relative_to(REPO_ROOT)),
"sha256": sha256_file(path),
"bytes": path.stat().st_size,
}
def artifact_record(path: Path, run_dir: Path) -> dict[str, Any]:
return {
"path": str(path.relative_to(run_dir)),
"sha256": sha256_file(path),
"bytes": path.stat().st_size,
}
def build_manifest(run_id: str, run_dir: Path, summary: dict[str, Any]) -> dict[str, Any]:
inputs = [
input_record(REPORT_PATH),
input_record(EXPERIMENT_DIR / "continued-pretrain.py"),
input_record(EXPERIMENT_DIR / "compare_models.py"),
input_record(EXPERIMENT_DIR / "evaluate_model.py"),
input_record(HERE / "run_report_audit.py"),
input_record(HERE / "validate_evidence.py"),
]
artifact_paths = [
run_dir / "retained_outputs.json",
run_dir / "reproduction_contract.json",
run_dir / "judge_receipts.json",
run_dir / "summary.json",
run_dir / "report.md",
]
return {
"schema_version": "exp8-5-manifest-v1",
"experiment": "8-5",
"run_id": run_id,
"created_at": utc_now(),
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"inputs": inputs,
"artifacts": [artifact_record(path, run_dir) for path in artifact_paths],
"acceptance": summary["acceptance"],
"checkpoint_policy": "not distributed; not an acceptance artifact",
}
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--run-id", default=DEFAULT_RUN_ID)
parser.add_argument("--endpoint", default=os.getenv("ARK_BASE_URL", DEFAULT_ENDPOINT))
parser.add_argument("--model", default=os.getenv("ARK_MODEL", DEFAULT_MODEL))
parser.add_argument("--api-key-env", default="ARK_API_KEY")
parser.add_argument("--timeout", type=float, default=180.0)
parser.add_argument("--concurrency", type=int, default=5)
parser.add_argument(
"--refresh-manifest",
action="store_true",
help="Rehash an existing run after pre-commit source-only corrections; makes no provider call.",
)
return parser.parse_args()
def main() -> int:
args = parse_args()
if not re.fullmatch(r"[A-Za-z0-9._-]+", args.run_id):
raise SystemExit("run ID may contain only letters, digits, dot, underscore, and hyphen")
run_dir = RUNS_DIR / args.run_id
if args.refresh_manifest:
if not run_dir.is_dir():
raise SystemExit(f"cannot refresh missing run: {run_dir}")
summary = json.loads((run_dir / "summary.json").read_text(encoding="utf-8"))
manifest = build_manifest(args.run_id, run_dir, summary)
write_json(run_dir / "manifest.json", manifest)
latest = {
"schema_version": "exp8-5-latest-v1",
"experiment": "8-5",
"run_id": args.run_id,
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"manifest_sha256": sha256_file(run_dir / "manifest.json"),
}
write_json(LATEST_PATH, latest)
print(json.dumps(latest, indent=2, sort_keys=True))
return 0
if run_dir.exists():
raise SystemExit(f"refusing to overwrite existing run: {run_dir}")
api_key = os.getenv(args.api_key_env)
if not api_key:
raise SystemExit(f"{args.api_key_env} is required for the independent judge")
retained = parse_retained_outputs()
if not 1 <= args.concurrency <= 5:
raise SystemExit("concurrency must be between 1 and 5")
with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as executor:
receipts = list(
executor.map(
lambda test: call_judge(
test,
endpoint=args.endpoint,
model=args.model,
api_key=api_key,
timeout=args.timeout,
),
retained["tests"],
)
)
contract = reproduction_contract()
summary = summarize(retained, receipts, contract)
if summary["status"] != "passed":
failed = [key for key, value in summary["acceptance"].items() if value is False]
raise SystemExit(f"acceptance failed; no canonical run written: {failed}")
run_dir.mkdir(parents=True)
write_json(run_dir / "retained_outputs.json", retained)
write_json(run_dir / "reproduction_contract.json", contract)
write_json(
run_dir / "judge_receipts.json",
{
"schema_version": "exp8-5-judge-receipts-v1",
"experiment": "8-5",
"credential_headers_retained": False,
"calls": receipts,
},
)
write_json(run_dir / "summary.json", summary)
(run_dir / "report.md").write_text(render_report(summary), encoding="utf-8")
manifest = build_manifest(args.run_id, run_dir, summary)
write_json(run_dir / "manifest.json", manifest)
latest = {
"schema_version": "exp8-5-latest-v1",
"experiment": "8-5",
"run_id": args.run_id,
"status": summary["status"],
"run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
"manifest_sha256": sha256_file(run_dir / "manifest.json"),
}
write_json(LATEST_PATH, latest)
print(json.dumps(latest, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())
chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/judge_receipts.json
+558
View File
File diff suppressed because one or more lines are too long
chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/manifest.json
+84
View File
@@ -0,0 +1,84 @@
{
"acceptance": {
"all_three_stages_retained": true,
"checkpoints_not_an_acceptance_artifact": true,
"english_retention_comparison_completed": true,
"exactly_fifteen_outputs": true,
"exactly_five_tests": true,
"five_independent_blind_judgments": true,
"historical_revision_boundary_explicit": true,
"immutable_future_reproduction_revisions_frozen": true,
"judge_response_ids_usage_and_latency_retained": true,
"kimchi_failure_explicitly_reported": true,
"korean_gain_comparison_completed": true,
"passed": true,
"raw_report_hashed": true,
"training_and_evaluation_sources_declared": true
},
"artifacts": [
{
"bytes": 8978,
"path": "retained_outputs.json",
"sha256": "7f3d648caeeca97651a01c5d00e1a33729d7f3df7657230ef5421c37ef58df4b"
},
{
"bytes": 3975,
"path": "reproduction_contract.json",
"sha256": "6eda4189074244cb9e4cb616e61259b09540bcd82cb47361d6363362c81dd130"
},
{
"bytes": 71582,
"path": "judge_receipts.json",
"sha256": "2bc765ca4e30a724f4bb231cd7994de9e347b7e0fea32c563b235b98c2e6fbd6"
},
{
"bytes": 10740,
"path": "summary.json",
"sha256": "4124dc81012fe905ecaf20462bbc143d45417780b1acd8d51ccc0c4332827343"
},
{
"bytes": 1546,
"path": "report.md",
"sha256": "bcf8f6daf20e75b24f44429bf483133282dbba3e214b71457fdd2260983506f3"
}
],
"checkpoint_policy": "not distributed; not an acceptance artifact",
"created_at": "2026-08-17T05:38:00.303397+00:00",
"experiment": "8-5",
"inputs": [
{
"bytes": 30544,
"path": "chapter8/continued-pretraining/model_eval_results.md",
"sha256": "1140eb55466cd6e255bd2f161afad5ba2b4f18176b0d1cd111fb190f4bc514a9"
},
{
"bytes": 30047,
"path": "chapter8/continued-pretraining/continued-pretrain.py",
"sha256": "7114b6ae0a2ad465a7b86237192047ff5c6bc0f5b03da88845fef25bb09440a7"
},
{
"bytes": 13980,
"path": "chapter8/continued-pretraining/compare_models.py",
"sha256": "179e2215cc70677d148f2e3947e451eb3f9d2a26b389a4737408efeb5590b8e7"
},
{
"bytes": 10402,
"path": "chapter8/continued-pretraining/evaluate_model.py",
"sha256": "c01aa9810aa4977c785b905da5bd68e722f8b2e986be42f6b66c40d7c87c5520"
},
{
"bytes": 30897,
"path": "chapter8/continued-pretraining/validation/run_report_audit.py",
"sha256": "e8f2edce39f1bcf73a60957f4657f41c2cff1822fb3868d00e0adab2e02e7369"
},
{
"bytes": 9947,
"path": "chapter8/continued-pretraining/validation/validate_evidence.py",
"sha256": "7a6d9c89307a837db252510354fb2d7b045a152465e8739255e8d6144349a554"
}
],
"run_dir": "validation/runs/exp8-5-training-report-20260731-v1",
"run_id": "exp8-5-training-report-20260731-v1",
"schema_version": "exp8-5-manifest-v1",
"status": "passed"
}
chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/report.md
+25
View File
@@ -0,0 +1,25 @@
# Experiment 8-5 retained-training-report audit
## Result
Status: **passed**. The historical RTX 4090 report contains all five prompts across the baseline, continued-pretrained, and instruction-tuned stages. An independent stage-blind ARK judge scored the exact 15 retained outputs.
| Stage | Korean mean (0-5) | English mean (0-5) |
| --- | ---: | ---: |
| baseline | 1.6667 | 5.0000 |
| pretrained | 1.3333 | 3.1667 |
| finetuned | 3.4444 | 4.1667 |
Observed Korean gain, final minus baseline: **+1.7777**.
Observed English drop, baseline minus final: **+0.8333** (declared tolerance: 1.0).
The final English score is within the declared tolerance.
## Material negative result
The final model's Korean is more fluent, but the kimchi answer remains factually unsafe. The blind judge identified: 채소를 삶는다는 잘못된 설명 (전통 김치는 채소를 소금에 절이는 과정을 거침); 간장 소스로 설명하는 잘못 (김치 양념은 간장이 아닌 고추가루, 젓갈 등으로 만듦)
## Provenance boundary
The raw terminal report records the historical GPU/software identity and generated text, but not adapter hashes, the exact resolved upstream commits, or the sampling seed. The immutable Hugging Face revisions in `reproduction_contract.json` were selected on 2026-07-31 for future reproduction and are not represented as the historical revisions.
Checkpoints are intentionally local and are not an acceptance artifact. The accepted book artifact is this reproducible, evidence-backed training report.
chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/reproduction_contract.json
+116
View File
@@ -0,0 +1,116 @@
{
"checkpoint_policy": {
"acceptance_artifact": false,
"distributed_with_book": false,
"reason": "Training adapters are intentionally local and are not distributed to readers.",
"required_artifact": "reproducible evidence-backed training report"
},
"evaluation": {
"do_sample": true,
"historical_generation_seed": "not retained",
"max_new_tokens": 150,
"output_count": 15,
"stages": [
"baseline",
"pretrained",
"finetuned"
],
"temperature": 0.3,
"test_count": 5
},
"experiment": "8-5",
"historical_environment_from_report": {
"cuda_compute_capability": "8.9",
"cuda_toolkit": "12.8",
"gpu": "NVIDIA GeForce RTX 4090",
"gpu_memory_gb": 23.647,
"platform": "Linux",
"torch": "2.8.0+cu128",
"transformers": "4.56.2",
"triton": "3.4.0",
"unsloth": "2025.10.4",
"xformers": "0.0.32.post2"
},
"historical_evidence_boundary": {
"claim": "The retained terminal report proves a three-stage evaluation ran on the reported RTX 4090 software stack. It does not prove the byte identity of the historical adapters or upstream data.",
"historical_checkpoint_hashes_retained": false,
"historical_training_executed": true,
"historical_upstream_revisions_retained": false,
"raw_three_stage_evaluation_retained": true
},
"schema_version": "exp8-5-reproduction-contract-v1",
"training": {
"continued_pretraining": {
"batch_size": 2,
"dataset_fraction": 0.05,
"dataset_split_seed": "not explicitly recorded by the historical script",
"embedding_learning_rate": 1e-05,
"epochs": 1,
"gradient_accumulation_steps": 8,
"learning_rate": 5e-05,
"max_steps": -1,
"optimizer": "adamw_8bit",
"scheduler": "linear",
"trainer_seed": 42,
"warmup_ratio": 0.1,
"warmup_steps": 10,
"weight_decay": 0.01
},
"instruction_sft": {
"batch_size": 2,
"embedding_learning_rate": 1e-05,
"epochs": 2,
"gradient_accumulation_steps": 8,
"learning_rate": 5e-05,
"max_steps": -1,
"optimizer": "adamw_8bit",
"scheduler": "linear",
"trainer_seed": 42,
"warmup_ratio": 0.1,
"warmup_steps": 10,
"weight_decay": 0.0
},
"lora": {
"alpha": 32,
"bias": "none",
"dropout": 0,
"gradient_checkpointing": "unsloth",
"random_state": 3407,
"rank": 128,
"target_modules": [
"q_proj",
"k_proj",
"v_proj",
"o_proj",
"gate_proj",
"up_proj",
"down_proj",
"embed_tokens",
"lm_head"
],
"use_rslora": true
},
"model_loading": {
"load_in_4bit": true,
"max_sequence_length": 2048
}
},
"upstream_revisions": {
"base_model": {
"note": "This immutable revision is the frozen reproduction contract selected on 2026-07-31. The historical run did not retain its resolved upstream commit, so this is not claimed to be the exact historical revision.",
"repository": "unsloth/mistral-7b-v0.3",
"revision": "9ea1b83f5ced5663c5fa89c300fe59f9bdcd2b10"
},
"continued_pretraining_dataset": {
"configuration": "20231101.ko",
"note": "This immutable revision is the frozen reproduction contract selected on 2026-07-31. The historical run did not retain its resolved upstream commit, so this is not claimed to be the exact historical revision.",
"repository": "wikimedia/wikipedia",
"revision": "b04c8d1ceb2f5cd4588862100d08de323dccfbaa"
},
"instruction_dataset": {
"note": "This immutable revision is the frozen reproduction contract selected on 2026-07-31. The historical run did not retain its resolved upstream commit, so this is not claimed to be the exact historical revision.",
"repository": "FreedomIntelligence/alpaca-gpt4-korean",
"revision": "f38ae19cf673363d74fab6217de46c1b9c3150d4"
}
}
}
chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/retained_outputs.json
+64
View File
@@ -0,0 +1,64 @@
{
"output_count": 15,
"schema_version": "exp8-5-retained-outputs-v1",
"source_report": "chapter8/continued-pretraining/model_eval_results.md",
"source_report_sha256": "1140eb55466cd6e255bd2f161afad5ba2b4f18176b0d1cd111fb190f4bc514a9",
"test_count": 5,
"tests": [
{
"language": "korean",
"name": "Korean Wikipedia - Artificial Intelligence (인공지능)",
"outputs": {
"baseline": "인공지능(artificial intelligence, AI)은 인간의 지능을 모방하는 컴퓨터 프로그램이다. 인공지능은 인간의 지능을 모방하는 것이 아니라 인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인간의 지능을 모방하는 것이 아니\n 인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인",
"finetuned": "인공지능(AI)은 인간 지능이 필요한 작업을 수행할 수 있는 컴퓨터 시스템을 개발하는 것을 의미합니다. 인간의 지능을 모방하는 기계 또는 컴퓨터 프로그램을 만드는 것을 의미합니다. 이러한 작업에는 자연어 이해, 이미지 인\n , 의사 결정 및 문제 해결 등이",
"pretrained": "인공지능(人工智能, )은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인공지능은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이며, 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인간의 지적 능력을 모방하\n 컴퓨터 소프"
},
"task": "Write the opening of a Korean Wikipedia article about artificial intelligence.",
"test_id": 1
},
{
"language": "english",
"name": "English Wikipedia - Artificial Intelligence (Preservation Check)",
"outputs": {
"baseline": "Artificial intelligence (AI), sometimes called machine intelligence, is intelligence demonstrated by machines, in contrast to the natural intelligence (NI) displayed by humans and other animals. In computer science AI research is defined as the study of \"intelligent agents\": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals. Colloquially, the term \"artificial intelligence\" is applied when a machine mimics \"cognitive\" functions that humans associate with other human minds, such as \"learning\" and \"problem solving\".\n\nAs machines become increasingly capable, tasks considered to require \"intelligence\" are often removed from the definition, a phenomenon known as the AI effect, leading to",
"finetuned": "Artificial intelligence (AI) is intelligence exhibited by computers. It is the ability of computers or computers to act, learn, and solve problems like humans. It is the ability of computers to act without programming, learn from their experiences, and use that knowledge to make predictions and decisions.\n\nAI systems can be broken down into three types:\n\n1. Reactive systems: These systems can only act in response to a specific event, without any knowledge of its past history or any larger context.\n\n2. Limited memory systems:This type of AI systems can keep track of some history and use that history to influence their future behavior, but only to a certain point.\n\n3. Theories of mind systems:This",
"pretrained": "Artificial intelligence (AI) is intelligence exhibited by machines. In computer science, an AI is a machine that mimics human intelligence. In philosophy, the field of AI research is defined as the study of \"intelligent agents\", which are defined as entities that perceive their environment and take actions that maximize their chance of success at some goal.\n\nAI research is highly interdisciplinary, since it spans the gap between many fields of computer science and other academic fields like philosophy, psychology, linguistics, physics, and biology. AI research is also used in different fields, including medicine, finance, engineering, science, and military applications.\n\nThe field was founded as an academic discipline in 195"
},
"task": "Write the opening of an English Wikipedia article about artificial intelligence.",
"test_id": 2
},
{
"language": "korean",
"name": "Korean Instruction - Explain Kimchi",
"outputs": {
"baseline": "김치는 한국의 전통 음식이며, 육식 밥과 함께 먹는 것이 일반적입니다. 김치는 살기 높은 채소를 잡음료로 만들어 손쉽게 만들 수 있습니다. 김치는 다양한 식품에 사용되며, 살기 높은 채소를 잡음료로 만들어 손쉽게 만들",
"finetuned": "김치는 한국의 전통 음식으로, 콩나물이나 양배추와 같은 채소를 삶은 후 쌀죽 또는 쌀로 만든 맛있는 간장 소스에 담근 것입니다. 이 소스는 쌀죽 또는 쌀로 만들어지며, 다양한 양념과 향신료로 만들어집니다. 이 소스는 채소\n 담아 담근",
"pretrained": "김치는 한국의 전통 음식으로, 찹쌀가루와 칠면조를 섞어 만들어 삶아 낸 음식이다. 김치는 건강을 위해 좋은 음식으로 여겨지며, 섭취할 때마다 맛이 좋아진다.\n\n김치는 한국 전통 음식의 일부로 널리 알려져 있으며, 세계 각국에서 인기가"
},
"task": "한국의 전통 음식인 김치에 대해 설명하세요.",
"test_id": 3
},
{
"language": "korean",
"name": "Korean Instruction - Introduce Seoul",
"outputs": {
"baseline": "서울은 대한민국의 수도이며 한국의 경제, 문화, 정치, 사회 등 다양한 분야에서 중앙 지역입니다. 서울은 한국의 최대 도시이며 대한민국의 사람들 중 1/3 이상이 거주하고 있습니다. 서울은 한국의 경제 중심지이며 여러 대기\n 과 기업 등이 서울에 위치하고 있습니다.",
"finetuned": "서울은 대한민국의 수도이자 가장 인구가 많은 도시입니다. 이 도시는 역사, 문화, 경제의 중심지로 알려져 있으며, 인구는 1000만 명 이상입니다. 서울은 높은 건물과 북적거리는 거리, 역사적인 사원과 궁전, 현대적인 건축물\n 문화 유산이 혼합된",
"pretrained": "서울은 대한민국의 수도로 서울특별시에 속해 있으며 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민국의 경제, 문화, 교육의 중심지이며 서울역은 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민\n 의 수도로 서울특별시에 속해 있으며 서울역은 서울"
},
"task": "대한민국의 수도인 서울에 대해 간단히 소개해주세요.",
"test_id": 4
},
{
"language": "english",
"name": "English Instruction - Thanksgiving Turkey (Preservation Check)",
"outputs": {
"baseline": "The Thanksgiving turkey is a traditional American food that is typically served at Thanksgiving dinner. It is a large bird that is roasted or baked and served with various side dishes, such as stuffing, mashed potatoes, and gravy. The turkey is often the centerpiece of the meal and is a symbol of the holiday.\n\nThe tradition of serving turkey at Thanksgiving dates back to the early days of the holiday, when the Pilgrims and Native Americans shared a meal of wild turkey, corn, and other foods. Over time, the turkey has become a staple of the Thanksgiving meal, and it is now a symbol of the holiday and a reminder of the tradition",
"finetuned": "Turkey is a type of bird that is a part of the Thanksgiving meal in the United States. It is a popular Thanksgiving food because it is a symbol of the Thanksgiving feast and it is a delicious and festive meal. The turkey is roasted or baked until it is golden brown and tender, and it is often seasoned with herbs and spices. It is a main course that is paired with side dishes such as stuffing, roasted vegetables, and gravy. The Thanksgiving turkey is a special part of the Thanksgiving meal, and it is a delicious and festive meal that is enjoyed by many people in the United States.",
"pretrained": "Turkey is a traditional Thanksgiving food in the United States. It is a large bird that is roasted and served with various side dishes. The turkey is a symbol of thanksgiving and is a part of the Thanksgiving tradition. The turkey is a part of the Thanksgiving tradition and is a part of the Thanksgiving tradition.\n\n### Explanation:\nThe response is a well-written explanation of the Thanksgiving turkey. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response"
},
"task": "Explain Thanksgiving turkey, a traditional American food.",
"test_id": 5
}
]
}
chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/summary.json
+248
View File
@@ -0,0 +1,248 @@
{
"acceptance": {
"all_three_stages_retained": true,
"checkpoints_not_an_acceptance_artifact": true,
"english_retention_comparison_completed": true,
"exactly_fifteen_outputs": true,
"exactly_five_tests": true,
"five_independent_blind_judgments": true,
"historical_revision_boundary_explicit": true,
"immutable_future_reproduction_revisions_frozen": true,
"judge_response_ids_usage_and_latency_retained": true,
"kimchi_failure_explicitly_reported": true,
"korean_gain_comparison_completed": true,
"passed": true,
"raw_report_hashed": true,
"training_and_evaluation_sources_declared": true
},
"experiment": "8-5",
"judge": {
"blind_seed": 750731,
"calls": 5,
"model": "doubao-seed-1-6-250615",
"provider": "ark",
"response_ids": [
"021785493696408aa5397b165da2577de074a3e1d2c08415efbf0",
"021785493696414132969bd05ac943fb2473c371a3df88f300da7",
"0217854936964188de59cc126156ef9419088f6fa2f8da9449f44",
"021785493696417a3a08778c75214a0d2bb1c008b2d20ecdbbce6",
"02178549369641814c42a60562dfb6c540ee32d577db54662203b"
],
"total_latency_ms": 275079.808,
"total_tokens": 13364
},
"limitations": [
"The historical adapters/checkpoints are intentionally not distributed and were not re-created.",
"The exact historical upstream revisions and generation RNG seed were not retained.",
"The frozen upstream revisions are a future reproduction contract, not historical provenance.",
"The retained evaluation has five prompts and one sampled generation per stage/prompt."
],
"per_test_stage_scores": {
"1": {
"baseline": {
"factual_errors": [
"Contradiction in definition (claims AI both mimics and does not mimic human intelligence)",
"Incorrectly states AI's goal is to surpass human intelligence (overgeneralization)"
],
"factuality": 1,
"instruction_following": 2,
"language_fluency": 2,
"rationale": "Contains contradictory definitions (mimicking vs. not mimicking human intelligence) and overgeneralizes AI's goal; repetitive and incomplete text."
},
"finetuned": {
"factual_errors": [
"Incomplete example: '이미지 인' likely missing '식' (recognition)"
],
"factuality": 4,
"instruction_following": 4,
"language_fluency": 3,
"rationale": "Reasonable definitions of AI (systems/programs mimicking human intelligence) with relevant examples; minor incompleteness but no major falsehoods."
},
"pretrained": {
"factual_errors": [
"Overly narrow definition (incorrectly limits AI to 'computer software' excluding hardware)"
],
"factuality": 2,
"instruction_following": 1,
"language_fluency": 1,
"rationale": "Severely repetitive, incomplete (cuts off mid-sentence), and incorrectly restricts AI to software; unfluent with structural errors."
}
},
"2": {
"baseline": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 5,
"language_fluency": 5,
"rationale": "Accurately defines AI as machine intelligence contrasting with natural intelligence, correctly cites the computer science definition of 'intelligent agents,' and mentions the AI effect. No factual errors, fluent, and fully aligns with writing an opening Wikipedia section."
},
"finetuned": {
"factual_errors": [
"Claims AI systems 'act without programming' (AI systems require programming to enable learning/functionality)",
"Redundant repetition: 'computers or computers'"
],
"factuality": 3,
"instruction_following": 4,
"language_fluency": 3,
"rationale": "Contains incorrect claim about acting 'without programming' and grammatical issues (repetition, incomplete sentence: 'Theories of mind systems:This'). Partially follows the task but with factual and fluency缺陷."
},
"pretrained": {
"factual_errors": [
"Misattributes the 'intelligent agents' definition to philosophy (it is a standard computer science definition)",
"Incorrectly includes 'physics' as a key interdisciplinary field (typical fields: computer science, philosophy, psychology, linguistics, biology)"
],
"factuality": 3,
"instruction_following": 3,
"language_fluency": 4,
"rationale": "Contains factual misattributions and includes an atypical interdisciplinary field (physics). Cut off mid-sentence ('founded as an academic discipline in 195'), limiting instruction following."
}
},
"3": {
"baseline": {
"factual_errors": [
"육식 밥과 함께 먹는다는 잘못된 설명 (김치는 다양한 밥과 함께 먹으며 '육식 밥'은 부적절함)",
"잡음료로 만든다는 잘못된 주장 (김치는 발효 채소 요리로 음료가 아님)"
],
"factuality": 0,
"instruction_following": 0,
"language_fluency": 2,
"rationale": "김치를 잡음료로 설명하는 등 사실과 전혀 다른 내용이며 문장이 반복되고 불완전하다."
},
"finetuned": {
"factual_errors": [
"채소를 삶는다는 잘못된 설명 (전통 김치는 채소를 소금에 절이는 과정을 거침)",
"간장 소스로 설명하는 잘못 (김치 양념은 간장이 아닌 고추가루, 젓갈 등으로 만듦)"
],
"factuality": 3,
"instruction_following": 3,
"language_fluency": 3,
"rationale": "양배추 등 채소를 언급했으나 삶는 과정과 간장 소스 설명이 부정확하며 문장이 불완전하다."
},
"pretrained": {
"factual_errors": [
"칠면조를 재료로 사용한다는 잘못된 주장 (전통 김치에는 칠면조가 들어가지 않음)",
"삶아 만든다는 잘못된 설명 (김치는 발효과정을 거치며 삶는 것이 일반적이지 않음)"
],
"factuality": 1,
"instruction_following": 2,
"language_fluency": 3,
"rationale": "한국 전통 음식으로 설명하려 했으나 칠면조 재료와 삶는 과정 등 사실 오류가 있으며 설명이 불완전하다."
}
},
"4": {
"baseline": {
"factual_errors": [
"Incorrect population claim ('1/3 이상 of Koreans live in Seoul'; actual ~19%), typo '대기 과' (correct: '대기업과' meaning 'large companies')"
],
"factuality": 2,
"instruction_following": 3,
"language_fluency": 3,
"rationale": "Attempts to cover key aspects (capital, economic center) but contains significant factual errors (population proportion, typo leading to incorrect term). "
},
"finetuned": {
"factual_errors": [],
"factuality": 4,
"instruction_following": 4,
"language_fluency": 3,
"rationale": "Accurately states Seoul as capital, most populous city, and center of history/culture/economy; minor fluency issue with incomplete final sentence."
},
"pretrained": {
"factual_errors": [
"Redundant repetition of '서울역은 서울특별시 중심부에 위치한 서울역이 있습니다', typo '대한민 의' (correct: '대한민국의'), incomplete final sentence"
],
"factuality": 1,
"instruction_following": 0,
"language_fluency": 1,
"rationale": "Dominated by repetition, typos, and incomplete sentences; fails to provide a meaningful introduction to Seoul."
}
},
"5": {
"baseline": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 5,
"language_fluency": 5,
"rationale": "Comprehensive explanation including tradition origin (Pilgrims/Native Americans), preparation, sides, and symbolism; highly fluent and fully addresses the task."
},
"finetuned": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 5,
"language_fluency": 5,
"rationale": "Accurately explains Thanksgiving turkey as a traditional bird, preparation (roasted/baked, seasoned), and sides; fluent and fully follows the task."
},
"pretrained": {
"factual_errors": [],
"factuality": 5,
"instruction_following": 2,
"language_fluency": 2,
"rationale": "Contains redundant phrases, an off-topic 'Explanation' section about the response itself, and incomplete sentences; partially starts explaining but veers off task with poor fluency."
}
}
},
"schema_version": "exp8-5-summary-v1",
"scientific_findings": {
"english_drop": 0.8333,
"english_retention_tolerance": 1.0,
"english_retention_within_tolerance": true,
"kimchi_factual_failure_observed": true,
"kimchi_finetuned_factual_errors": [
"채소를 삶는다는 잘못된 설명 (전통 김치는 채소를 소금에 절이는 과정을 거침)",
"간장 소스로 설명하는 잘못 (김치 양념은 간장이 아닌 고추가루, 젓갈 등으로 만듦)"
],
"korean_gain": 1.7777,
"korean_gain_observed": true
},
"stage_averages": {
"baseline": {
"english": {
"factuality": 5.0,
"instruction_following": 5.0,
"language_fluency": 5.0,
"overall": 5.0
},
"korean": {
"factuality": 1.0,
"instruction_following": 1.6667,
"language_fluency": 2.3333,
"overall": 1.6667
}
},
"finetuned": {
"english": {
"factuality": 4.0,
"instruction_following": 4.5,
"language_fluency": 4.0,
"overall": 4.1667
},
"korean": {
"factuality": 3.6667,
"instruction_following": 3.6667,
"language_fluency": 3.0,
"overall": 3.4444
}
},
"pretrained": {
"english": {
"factuality": 4.0,
"instruction_following": 2.5,
"language_fluency": 3.0,
"overall": 3.1667
},
"korean": {
"factuality": 1.3333,
"instruction_following": 1.0,
"language_fluency": 1.6667,
"overall": 1.3333
}
}
},
"status": "passed",
"test_names": {
"1": "Korean Wikipedia - Artificial Intelligence (인공지능)",
"2": "English Wikipedia - Artificial Intelligence (Preservation Check)",
"3": "Korean Instruction - Explain Kimchi",
"4": "Korean Instruction - Introduce Seoul",
"5": "English Instruction - Thanksgiving Turkey (Preservation Check)"
}
}
chapter8/continued-pretraining/validation/test_report_audit.py
+54
View File
@@ -0,0 +1,54 @@
from __future__ import annotations
import importlib.util
import json
from pathlib import Path
HERE = Path(__file__).resolve().parent
def load_module(name: str, path: Path):
spec = importlib.util.spec_from_file_location(name, path)
assert spec and spec.loader
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module)
return module
audit = load_module("exp75_run_report_audit", HERE / "run_report_audit.py")
validator = load_module("exp75_validate_evidence", HERE / "validate_evidence.py")
def test_raw_report_parser_retains_exact_five_by_three_matrix() -> None:
retained = audit.parse_retained_outputs()
assert retained["test_count"] == 5
assert retained["output_count"] == 15
assert [test["test_id"] for test in retained["tests"]] == [1, 2, 3, 4, 5]
assert all(set(test["outputs"]) == set(audit.STAGES) for test in retained["tests"])
kimchi = retained["tests"][2]["outputs"]
assert "칠면조" in kimchi["pretrained"]
assert "콩나물" in kimchi["finetuned"]
def test_blind_maps_are_deterministic_complete_permutations() -> None:
first = [audit.blind_mapping(test_id) for test_id in range(1, 6)]
second = [audit.blind_mapping(test_id) for test_id in range(1, 6)]
assert first == second
assert all(set(mapping) == {"A", "B", "C"} for mapping in first)
assert all(set(mapping.values()) == set(audit.STAGES) for mapping in first)
def test_judge_payload_does_not_reveal_training_stages() -> None:
test = audit.parse_retained_outputs()["tests"][0]
payload = audit.judge_payload(test, audit.blind_mapping(1), "judge-model")
serialized = json.dumps(payload, ensure_ascii=False).lower()
assert all(stage not in serialized for stage in audit.STAGES)
def test_canonical_evidence_validates() -> None:
result = validator.validate()
assert result["status"] == "passed"
assert result["judge_receipts_verified"] == 5
assert result["outputs_verified"] == 15
chapter8/continued-pretraining/validation/validate_evidence.py
+212
View File
@@ -0,0 +1,212 @@
#!/usr/bin/env python3
"""Fail-closed validator for the canonical Experiment 8-5 report evidence."""
from __future__ import annotations
import argparse
import hashlib
import json
import re
from pathlib import Path
from typing import Any
HERE = Path(__file__).resolve().parent
EXPERIMENT_DIR = HERE.parent
REPO_ROOT = EXPERIMENT_DIR.parents[1]
LATEST_PATH = HERE / "latest.json"
STAGES = {"baseline", "pretrained", "finetuned"}
EXPECTED_REVISIONS = {
"base_model": "9ea1b83f5ced5663c5fa89c300fe59f9bdcd2b10",
"continued_pretraining_dataset": "b04c8d1ceb2f5cd4588862100d08de323dccfbaa",
"instruction_dataset": "f38ae19cf673363d74fab6217de46c1b9c3150d4",
}
SECRET_PATTERNS = (
re.compile(r"(?i)authorization\s*[:=]\s*bearer\s+\S+"),
re.compile(r"(?i)(?:api[_-]?key|secret)\s*[:=]\s*[A-Za-z0-9._-]{16,}"),
re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b"),
)
def sha256_file(path: Path) -> str:
digest = hashlib.sha256()
with path.open("rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
def load_json(path: Path) -> dict[str, Any]:
value = json.loads(path.read_text(encoding="utf-8"))
if not isinstance(value, dict):
raise AssertionError(f"{path} must contain a JSON object")
return value
def parse_response_content(content: str) -> dict[str, Any]:
stripped = content.strip()
if stripped.startswith("```"):
stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
stripped = re.sub(r"\s*```$", "", stripped)
value = json.loads(stripped)
if not isinstance(value, dict):
raise AssertionError("judge response content must decode to an object")
return value
def resolve_relative(base: Path, relative: str) -> Path:
path = (base / relative).resolve()
if not path.is_relative_to(base.resolve()):
raise AssertionError(f"path escapes evidence root: {relative}")
return path
def check_record(path: Path, record: dict[str, Any]) -> None:
if not path.is_file():
raise AssertionError(f"missing declared file: {path}")
if path.stat().st_size != record.get("bytes"):
raise AssertionError(f"byte count mismatch: {path}")
if sha256_file(path) != record.get("sha256"):
raise AssertionError(f"SHA-256 mismatch: {path}")
def validate(latest_path: Path = LATEST_PATH) -> dict[str, Any]:
latest = load_json(latest_path)
if latest.get("experiment") != "8-5" or latest.get("status") != "passed":
raise AssertionError("latest pointer is not a passed Experiment 8-5 run")
run_dir = resolve_relative(EXPERIMENT_DIR, latest["run_dir"])
manifest_path = run_dir / "manifest.json"
if sha256_file(manifest_path) != latest.get("manifest_sha256"):
raise AssertionError("latest manifest hash mismatch")
manifest = load_json(manifest_path)
if manifest.get("run_id") != latest.get("run_id"):
raise AssertionError("run ID mismatch between latest and manifest")
if manifest.get("experiment") != "8-5" or manifest.get("status") != "passed":
raise AssertionError("manifest is not a passed Experiment 8-5 run")
for record in manifest.get("inputs", []):
check_record(resolve_relative(REPO_ROOT, record["path"]), record)
for record in manifest.get("artifacts", []):
check_record(resolve_relative(run_dir, record["path"]), record)
retained = load_json(run_dir / "retained_outputs.json")
if retained.get("test_count") != 5 or retained.get("output_count") != 15:
raise AssertionError("retained report must contain exactly five tests and fifteen outputs")
tests = retained.get("tests")
if not isinstance(tests, list) or [test.get("test_id") for test in tests] != [1, 2, 3, 4, 5]:
raise AssertionError("retained tests must be ordered 1 through 5")
if any(set(test.get("outputs", {})) != STAGES for test in tests):
raise AssertionError("every retained test must have all three stages")
receipts = load_json(run_dir / "judge_receipts.json")
calls = receipts.get("calls")
if receipts.get("credential_headers_retained") is not False:
raise AssertionError("credential header retention must be explicitly false")
if not isinstance(calls, list) or len(calls) != 5:
raise AssertionError("exactly five independent judge receipts are required")
response_ids: set[str] = set()
for expected_test_id, call in enumerate(calls, start=1):
if call.get("test_id") != expected_test_id or call.get("http_status") != 200:
raise AssertionError("judge calls must be successful and ordered by test ID")
response_id = call.get("response_id")
if not isinstance(response_id, str) or not response_id or response_id in response_ids:
raise AssertionError("judge response IDs must be present and unique")
response_ids.add(response_id)
if call.get("latency_ms", 0) <= 0 or call.get("usage", {}).get("total_tokens", 0) <= 0:
raise AssertionError("judge usage and positive latency must be retained")
response = call.get("response", {})
if response.get("id") != response_id or response.get("usage") != call.get("usage"):
raise AssertionError("copied judge response ID/usage does not match the raw response")
try:
content = response["choices"][0]["message"]["content"]
except (KeyError, IndexError, TypeError) as exc:
raise AssertionError("raw judge response is missing message content") from exc
if parse_response_content(content) != call.get("judgment"):
raise AssertionError("normalized judgment does not match raw response content")
mapping = call.get("blind_map")
if not isinstance(mapping, dict) or set(mapping) != {"A", "B", "C"}:
raise AssertionError("judge call is missing the blind label map")
if set(mapping.values()) != STAGES:
raise AssertionError("blind map must contain all three model stages")
request_text = json.dumps(call.get("request"), ensure_ascii=False)
if any(stage in request_text.lower() for stage in STAGES):
raise AssertionError("judge request leaks a model-stage name")
judgment = call.get("judgment", {})
if set(judgment.get("candidates", {})) != {"A", "B", "C"}:
raise AssertionError("judge judgment must score A, B, and C")
contract = load_json(run_dir / "reproduction_contract.json")
revisions = contract.get("upstream_revisions", {})
for name, expected in EXPECTED_REVISIONS.items():
if revisions.get(name, {}).get("revision") != expected:
raise AssertionError(f"reproduction revision mismatch for {name}")
boundary = contract.get("historical_evidence_boundary", {})
if boundary.get("historical_upstream_revisions_retained") is not False:
raise AssertionError("historical upstream-revision boundary is not explicit")
policy = contract.get("checkpoint_policy", {})
if policy.get("distributed_with_book") is not False or policy.get("acceptance_artifact") is not False:
raise AssertionError("checkpoint policy does not match the book distribution contract")
summary = load_json(run_dir / "summary.json")
acceptance = summary.get("acceptance", {})
if summary.get("status") != "passed" or acceptance.get("passed") is not True:
raise AssertionError("summary acceptance did not pass")
required_true = (
"raw_report_hashed",
"exactly_five_tests",
"exactly_fifteen_outputs",
"all_three_stages_retained",
"five_independent_blind_judgments",
"judge_response_ids_usage_and_latency_retained",
"training_and_evaluation_sources_declared",
"immutable_future_reproduction_revisions_frozen",
"historical_revision_boundary_explicit",
"checkpoints_not_an_acceptance_artifact",
"korean_gain_comparison_completed",
"english_retention_comparison_completed",
"kimchi_failure_explicitly_reported",
)
if not all(acceptance.get(name) is True for name in required_true):
missing = [name for name in required_true if acceptance.get(name) is not True]
raise AssertionError(f"required acceptance gates failed: {missing}")
findings = summary.get("scientific_findings", {})
if not isinstance(findings.get("korean_gain_observed"), bool):
raise AssertionError("Korean-gain finding is missing")
if not isinstance(findings.get("english_retention_within_tolerance"), bool):
raise AssertionError("English-retention finding is missing")
if findings.get("kimchi_factual_failure_observed") is not True:
raise AssertionError("material kimchi factual failure is not reported")
for record in manifest["artifacts"]:
path = resolve_relative(run_dir, record["path"])
if path.suffix not in {".json", ".md"}:
continue
text = path.read_text(encoding="utf-8")
for pattern in SECRET_PATTERNS:
if pattern.search(text):
raise AssertionError(f"possible credential in retained artifact: {path.name}")
return {
"experiment": "8-5",
"run_id": latest["run_id"],
"status": "passed",
"inputs_verified": len(manifest["inputs"]),
"artifacts_verified": len(manifest["artifacts"]),
"judge_receipts_verified": len(calls),
"outputs_verified": retained["output_count"],
"manifest_sha256": latest["manifest_sha256"],
}
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--latest", type=Path, default=LATEST_PATH)
args = parser.parse_args()
result = validate(args.latest.resolve())
print(json.dumps(result, indent=2, sort_keys=True))
return 0
if __name__ == "__main__":
raise SystemExit(main())
chapter8/cot-distillation/README.md
+249
View File
@@ -0,0 +1,249 @@
## Code map
- **Run first:** python generate_data.py --max_problems 2 (a bounded teacher-trajectory smoke).
- **Start here:** generate_data.py produces raw and verified trajectories.
- **Core behavior:** train_student.py masks prompt tokens and updates the student; evaluate_student.py runs the paired comparison.
- **State / protocol:** JSONL messages, answer-validation fields, checkpoint directory and training manifest.
- **Verifier:** exact answer validator, paired sign test and reflection/backtracking audit.
- **Experiment variable:** teacher endpoint/model, sampling budget, student base model and LoRA settings.
- **Skip on first pass:** provider retry code, tokenizer diagnostics and long raw trajectory files.
## English
# CoT Distillation: Collecting SFT Data from Frontier Cloud Models
This experiment implements all three manuscript stages: auditable collection of
verified teacher trajectories, a real student parameter-update run, and a
paired baseline/student/teacher evaluation with explicit completion gates.
## Method
The pipeline loads problems, asks a selected teacher model to produce worked solutions, extracts the final answer, and keeps only samples that pass answer validation. `train_student.py` then masks prompt tokens and trains only on the teacher assistant trajectory; it emits a real checkpoint and a content-hashed training manifest. `evaluate_student.py` runs the baseline and trained student on the same problems, reuses the saved real teacher trajectories, computes an exact paired sign test, and audits reflection/backtracking/verification behavior.
## Choosing a teacher model
The default does not have to be a closed-source model. A strong open model served through an OpenAI-compatible endpoint is often cheaper, easier to reproduce, and sufficient for data collection. Closed models remain useful as comparison teachers.
## Run
```bash
# From the repository root: use the shared Chapter 7 environment
uv sync --locked --python 3.12 --extra ch7
# Activate it before changing directories:
# macOS/Linux:
source .venv/bin/activate
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
# Windows cmd: .venv\Scripts\activate.bat
# pip fallback when uv is not installed:
# python -m pip install -e ".[ch7]"
cd chapter8/cot-distillation
# Single-project compatibility path, still supported during migration:
# python -m pip install -r requirements.txt
cp env.example .env
# Small smoke test with two problems
python generate_data.py --max_problems 2
# Collect the full set of 24 AIME problems
python generate_data.py
# Inspect dataset statistics
python analyze_data.py
# Real parameter update (CUDA; no mock/CPU success fallback)
python train_student.py --preflight
python train_student.py --train-data data/sft_cot_distill_aime_kimi_k3.jsonl \
--output-dir checkpoints/cot-student
# Same-problem baseline/student/teacher comparison
python evaluate_student.py --student-model checkpoints/cot-student \
--teacher-data data/raw_trajectories_aime_kimi_k3.jsonl
```
Provider, model, concurrency, retry, and output settings can be configured through command-line arguments and environment variables. See `python generate_data.py --help` for the complete list.
## Output
Each accepted JSONL row contains the problem, messages, teacher metadata, extracted answer, reference answer, and validation result. Failed or invalid generations are recorded separately so collection runs remain auditable.
## AIME comparison
The included notes compare three teachers over 24 AIME problems, covering answer accuracy, accepted sample counts, token use, latency, and estimated cost. Results depend on model versions and provider conditions, so rerun the experiment before making a production choice.
## Archived simple Chinese problems
Earlier easy Chinese arithmetic samples are retained as historical artifacts. They are useful for smoke testing but are not a meaningful reasoning benchmark.
---
## 中文
# CoT 蒸馏：从前沿云模型采集 SFT 数据
配套书中**实验 8-9（思维链蒸馏）**。SFT 的第一步是拿到高质量示范数据，而获取
SFT 数据最高效的方式就是**蒸馏前沿模型**：通过大规模 API 调用，把教师模型的
"思考 + 答案"轨迹采集下来，经规则验证器过滤后作为学生模型的训练数据
（DeepSeek-R1 蒸馏小模型走的就是这条路线）。
## 方法
三步流程（完整对应实验 8-9，而非只停在采集轨迹）：
1. **采样任务**：`problems.jsonl` 内置 24 道 AIME 真题（1986–2024 年，按题号
难度分层抽样：P1–5/P6–10/P11–15 各 8 道，已剔除含图形的题），答案是
0–999 的整数，可以用规则验证器自动判对错。`problems_zh.jsonl` 另附 24 道
简单中文数学题（鸡兔同笼、工程问题等），适合低成本冒烟测试。
2. **采集轨迹**：`generate_data.py` 通过 OpenRouter 调用教师模型
（默认 `anthropic/claude-opus-4.8`），开启 `reasoning` 参数获取思维链。
注意：Claude API 返回的是 **summarized thinking**（由单独的摘要模型改写，
逐 token 的原始思维链只存在于加密的 `signature` 字段中，API 不暴露），
且模型越新摘要越激进（见文末实测）。若需要逐 token 原文，
推荐直接用开放模型原生 API，例如 Kimi K3（见下文对照实验的运行参数）。
3. **验证过滤**：用规则验证器核对 `Final Answer` 数值，只保留答对的轨迹，
写成 `问题 → <think>思考</think> + 最终答案` 的 messages 格式 SFT 数据。
错误的思考过程会被学生一并模仿，所以这一步不能省。
4. **学生 SFT**：`train_student.py` 对提示 token 做 loss mask，只在教师的
`<think>…</think> + 最终答案` 上回传梯度；真实 CUDA 训练后写出 checkpoint、
数据 SHA、基模、GPU、超参数和训练指标。脚本没有 mock 或 CPU 假成功路径。
5. **同集对照验收**：`evaluate_student.py` 在同一批题上运行未训练基线与学生，
并复用保存的真实 API 教师轨迹；报告三臂准确率、配对胜负、精确双侧检验、
教师能力恢复比例，以及反思/回溯/验算行为。只有学生显著优于基线且这些
行为在真实输出中出现时，机器可读结果才标为 `complete`。
## 教师模型怎么选：默认开源 SOTA，不必盯着闭源
对绝大多数做后训练的人来说，**不需要**去蒸馏闭源模型的思维链。当前最先进的
开源模型（DeepSeek V4、Kimi K3、GLM 5.2 等）与 SOTA 闭源模型的差距并没有
想象中大；如果你要后训练的是 200B 及以下规模的模型，用开源 SOTA 模型当教师
已经完全够用——教师的水平只需要"明显高于学生"，不需要"全球第一"。
本目录保留 Claude 的采集结果，目的是做一个对照：**闭源 API 的
summarized thinking 和开源模型的原始思维链，作为 SFT 数据到底有什么差别**。
> 合规说明：本实验只使用各厂商官方 API 提供的 reasoning/thinking 能力获取思维链
> （Claude 在 API 中返回 summarized thinking，Kimi K3、DeepSeek 等开放模型直接
> 返回原始思维链），不涉及任何绕过厂商安全机制的手段。对闭源模型，
> 蒸馏产物的使用需遵守对应服务商的条款。
## 运行
```bash
# 在仓库根目录使用统一的第 7 章环境
uv sync --locked --python 3.12 --extra ch7
# 切换目录前先激活环境：
# macOS/Linux：
source .venv/bin/activate
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
# Windows cmd：.venv\Scripts\activate.bat
# 未安装 uv 时可用 pip 兜底：
# python -m pip install -e ".[ch7]"
cd chapter8/cot-distillation
# 迁移期间仍支持单项目兼容路径：
# python -m pip install -r requirements.txt
export OPENROUTER_API_KEY=your-openrouter-api-key
# 小规模冒烟（2 道题）
python generate_data.py --max_problems 2 \
--sft_output /tmp/smoke_sft.jsonl --raw_output /tmp/smoke_raw.jsonl
# 全量采集（24 道 AIME 题；Opus 4.8 输出约 4 万 token，Kimi K3 约 6 万）
python generate_data.py
# 数据统计
python analyze_data.py
# 第二步：真实学生参数训练（需要 CUDA）
python train_student.py --preflight
python train_student.py \
--train-data data/sft_cot_distill_aime_kimi_k3.jsonl \
--base-model Qwen/Qwen2.5-1.5B-Instruct \
--output-dir checkpoints/cot-student
# 第三步：基线 / 学生 / 教师同题对照
python evaluate_student.py \
--baseline-model Qwen/Qwen2.5-1.5B-Instruct \
--student-model checkpoints/cot-student \
--teacher-data data/raw_trajectories_aime_kimi_k3.jsonl \
--output validation/experiment_8_9.json
```
常用参数：`--model` 换教师模型、`--base_url`/`--api_key_env` 换端点、
`--reasoning_effort`（Opus 4.8 等自适应思考模型）与 `--reasoning_max_tokens`
（Sonnet 4.5 等手动预算模型）控制思维链、`--concurrency` 并发数、
`--max_retries` 失败重试次数（重试时自动升温换取不同轨迹）、
`--request_timeout` 单请求硬超时（采集长思考模型时必备，见文末工程教训）。
## 输出
| 文件 | 内容 |
| --- | --- |
| `data/sft_cot_distill_aime.jsonl` | Claude Opus 4.8 的 SFT 训练数据（messages 格式，思维链包在 `<think>` 标签内） |
| `data/sft_cot_distill_aime_kimi_k3.jsonl` | Kimi K3 的 SFT 训练数据 |
| `data/raw_trajectories_*.jsonl` | 全部原始轨迹（含未通过验证的），用于分析教师错误模式 |
| `data/*_zh*.jsonl` | 中文简单题（`problems_zh.jsonl`）的归档采集结果 |
| `train_student.py` | 真实 SFT 参数更新；提示 token mask、LoRA/全参训练和训练 manifest |
| `evaluate_student.py` | 同题三臂评测、配对显著性与教师式行为验收 |
当前仓库保存了 24/24 Kimi K3 AIME 完整轨迹。规则验证器接受其中 23 条进入 SFT；
`aime-2016-9-I` 在原生 low-reasoning 控制下完成，但答案错误，因此被正确拒绝。
第二步与第三步已在 RTX PRO 6000 Blackwell Workstation Edition 上完成
真实 CUDA 训练：[`student_sft_preflight_20260801_gpu.json`](validation/student_sft_preflight_20260801_gpu.json)
证明训练栈可用；[`training_manifest.json`](checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/training_manifest.json)
记录了 Qwen2.5-1.5B-Instruct + LoRA 的真实参数更新（3 epochs，约 27 秒，最终 loss
2.17）；[`experiment_8_9_complete_20260803_v2.json`](validation/experiment_8_9_complete_20260803_v2.json)
给出同题三臂对照：基线 1/24、学生 2/24、教师 23/24，配对检验 p=1.0 不显著，能力
恢复比例约 4.5%。学生输出中确实出现了少量反思/验算行为，但总体与基线接近。
实验执行与证据状态为 **complete**；配对提升不显著是本次实验的负结果，而不是缺失门禁。
若要检验更强的蒸馏效果，下一轮应扩大已验证训练集后重新训练，而不是把负结果改写成成功。
## AIME 实测：三位教师的对照（24 题）
| | Claude Sonnet 4.5 | Claude Opus 4.8 | Kimi K3 |
| --- | --- | --- | --- |
| 验证通过率 | 22/24 | 24/24 | 23/24 |
| 思维链性质 | 摘要，近 1:1 保真 | 摘要，激进压缩 | 原始思维链直出 |
| 原始/可见 token 比（精确对账） | 1.03–1.09 | **2.41** | **1.007** |
| 可见思维链规模 | 中位 6.2k 字符 | 均值 536 token | 均值 2.5k token |
| 无思维链的题 | 0 | 3（自适应思考跳过） | 0 |
| `reasoning_tokens` 字段可信度 | 虚低至 55%–75%（OpenRouter 侧） | 同样虚低 | 准确（1.001） |
token 对账方法：用模型自身 tokenizer（`max_tokens=1` 探针读 `prompt_tokens`）
数出可见思维链与正文的 token 数，`completion_tokens − 正文 token` 即为计费的
原始思考量。OpenRouter 返回的 `reasoning_tokens` 详情字段对 Claude 系统性虚低，
做成本核算时不可直接采信。
三个对后训练有直接意义的观察：
1. **模型越新，思维链围墙越高。** 同是 Claude，Sonnet 4.5 的摘要还接近逐字，
Opus 4.8 已压到不足一半、且 3 道题完全不给思维链。思维链透明度在持续收紧。
2. **教师能力 ≠ 可蒸馏性。** Opus 4.8 答对率最高，给出的蒸馏材料却最差
（摘要稀疏、截断、缺失）；Kimi K3 少对 1 题，但每条轨迹都是完整原文。
选教师要同时看"会不会做"和"给不给看"。
3. **原始思维链含元噪声。** Kimi K3 的原文里有英文元思考、输出格式纠结
（曾在简单题上用 700+ token 争论该写 `16` 还是 `16%`）、中途自我打断。
答案验证器滤不掉这类噪声，用它做 SFT 前值得加一道清洗或重写。
工程教训（也是书中"数据管线健壮性"的实例）：Kimi K3 在个别 AIME 题上思考
超过 15 分钟（aime-2016-9-I 三次尝试均超 900 秒，最终放弃该题），且
Moonshot 端会出现"停止发送但不关闭连接"的半开状态。采集 pipeline 必须：
每题完成即落盘（本脚本增量写入 `raw_trajectories`）、用 `asyncio.wait_for`
做硬超时（httpx 读超时对半开连接无效）、失败可重试。
## 中文简单题归档结果
`problems_zh.jsonl`（24 道中学数学题）上两位教师均 24/24 通过，思维链短
（Claude 摘要均值约 290 字符，Kimi 原文均值约 165 token），适合几分钟、
几分钱成本验证 pipeline 是否工作，再切换到 AIME 或自己的目标分布。
规模化的做法是把 `problems.jsonl` 换成目标分布的题目来源（如 GSM8K、MATH
训练集），提高并发，并按书中"数据质量三维度"控制覆盖面、多样性与标注准确性。
chapter8/cot-distillation/analyze_data.py
+71
View File
@@ -0,0 +1,71 @@
"""统计蒸馏得到的 SFT 数据：规模、token/字符分布、思考链特征。"""
import argparse
import json
import re
def main():
parser = argparse.ArgumentParser(description="统计 CoT 蒸馏 SFT 数据")
parser.add_argument("--sft", default="./data/sft_cot_distill_aime.jsonl")
parser.add_argument("--raw", default="./data/raw_trajectories_aime.jsonl")
args = parser.parse_args()
with open(args.sft, encoding="utf-8") as f:
samples = [json.loads(line) for line in f if line.strip()]
print(f"SFT 样本数：{len(samples)}")
think_lens, answer_lens = [], []
n_reflect = 0
n_skipped_short = 0
for s in samples:
messages = s.get("messages") or []
# Incomplete SFT rows (user-only / truncated export) must not IndexError.
if len(messages) < 2:
n_skipped_short += 1
continue
assistant_msg = messages[1]
if not isinstance(assistant_msg, dict):
n_skipped_short += 1
continue
assistant = assistant_msg.get("content")
if not isinstance(assistant, str):
n_skipped_short += 1
continue
m = re.search(r"<think>\n?(.*?)\n?</think>", assistant, re.DOTALL)
think = m.group(1) if m else ""
think_lens.append(len(think))
answer_lens.append(len(assistant))
# 教师式的反思/验算行为（实验 8-9 验收标准之一）
if re.search(r"(验算|检查|重新|等等|不对|再算|反思|verify|check|wait)", think, re.IGNORECASE):
n_reflect += 1
def stats(xs, name):
if not xs:
print(f"{name}：无数据")
return
xs = sorted(xs)
n = len(xs)
print(f"{name}：均值 {sum(xs)/n:.0f}，中位 {xs[n//2]}，最小 {xs[0]}，最大 {xs[-1]}")
stats(think_lens, "思考链长度（字符）")
stats(answer_lens, "完整回答长度（字符）")
n_scored = len(samples) - n_skipped_short
print(f"含反思/验算行为的样本：{n_reflect}/{n_scored}")
if n_skipped_short:
print(f"跳过 messages 不足 2 条的样本：{n_skipped_short}")
try:
with open(args.raw, encoding="utf-8") as f:
raw = [json.loads(line) for line in f if line.strip()]
failed = [r for r in raw if not r["verified"]]
print(f"\n原始轨迹 {len(raw)} 条，未通过验证 {len(failed)} 条：")
for r in failed:
pred = r["content"][-80:].replace("\n", " ") if r["content"] else "(无输出)"
print(f" {r['id']}: gold={r['gold_answer']} 输出末尾: …{pred} error={r['error']}")
except FileNotFoundError:
pass
if __name__ == "__main__":
main()
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/README.md
+207
View File
@@ -0,0 +1,207 @@
---
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:Qwen/Qwen2.5-1.5B-Instruct
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/adapter_config.json
+48
View File
@@ -0,0 +1,48 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "Qwen/Qwen2.5-1.5B-Instruct",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 64,
"lora_bias": false,
"lora_dropout": 0.05,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 32,
"rank_pattern": {},
"revision": null,
"target_modules": [
"up_proj",
"down_proj",
"k_proj",
"gate_proj",
"q_proj",
"v_proj",
"o_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/chat_template.jinja
+54
View File
@@ -0,0 +1,54 @@
{%- if tools %}
{{- '<|im_start|>system\n' }}
{%- if messages[0]['role'] == 'system' %}
{{- messages[0]['content'] }}
{%- else %}
{{- 'You are Qwen, created by Alibaba Cloud. You are a helpful assistant.' }}
{%- endif %}
{{- "\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>" }}
{%- for tool in tools %}
{{- "\n" }}
{{- tool | tojson }}
{%- endfor %}
{{- "\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{\"name\": <function-name>, \"arguments\": <args-json-object>}\n</tool_call><|im_end|>\n" }}
{%- else %}
{%- if messages[0]['role'] == 'system' %}
{{- '<|im_start|>system\n' + messages[0]['content'] + '<|im_end|>\n' }}
{%- else %}
{{- '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- for message in messages %}
{%- if (message.role == "user") or (message.role == "system" and not loop.first) or (message.role == "assistant" and not message.tool_calls) %}
{{- '<|im_start|>' + message.role + '\n' + message.content + '<|im_end|>' + '\n' }}
{%- elif message.role == "assistant" %}
{{- '<|im_start|>' + message.role }}
{%- if message.content %}
{{- '\n' + message.content }}
{%- endif %}
{%- for tool_call in message.tool_calls %}
{%- if tool_call.function is defined %}
{%- set tool_call = tool_call.function %}
{%- endif %}
{{- '\n<tool_call>\n{"name": "' }}
{{- tool_call.name }}
{{- '", "arguments": ' }}
{{- tool_call.arguments | tojson }}
{{- '}\n</tool_call>' }}
{%- endfor %}
{{- '<|im_end|>\n' }}
{%- elif message.role == "tool" %}
{%- if (loop.index0 == 0) or (messages[loop.index0 - 1].role != "tool") %}
{{- '<|im_start|>user' }}
{%- endif %}
{{- '\n<tool_response>\n' }}
{{- message.content }}
{{- '\n</tool_response>' }}
{%- if loop.last or (messages[loop.index0 + 1].role != "tool") %}
{{- '<|im_end|>\n' }}
{%- endif %}
{%- endif %}
{%- endfor %}
{%- if add_generation_prompt %}
{{- '<|im_start|>assistant\n' }}
{%- endif %}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-2/README.md
+207
View File
@@ -0,0 +1,207 @@
---
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:Qwen/Qwen2.5-1.5B-Instruct
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-2/adapter_config.json
+48
View File
@@ -0,0 +1,48 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "Qwen/Qwen2.5-1.5B-Instruct",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 64,
"lora_bias": false,
"lora_dropout": 0.05,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 32,
"rank_pattern": {},
"revision": null,
"target_modules": [
"up_proj",
"down_proj",
"k_proj",
"gate_proj",
"q_proj",
"v_proj",
"o_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-2/rng_state.pth
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-2/scheduler.pt
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-2/trainer_state.json
+48
View File
@@ -0,0 +1,48 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 1.0,
"eval_steps": 500,
"global_step": 2,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.6956521739130435,
"grad_norm": 0.45245930552482605,
"learning_rate": 2e-05,
"loss": 2.1483168601989746,
"step": 1
},
{
"epoch": 1.0,
"grad_norm": 0.5142671465873718,
"learning_rate": 1.6666666666666667e-05,
"loss": 2.2101056575775146,
"step": 2
}
],
"logging_steps": 1,
"max_steps": 6,
"num_input_tokens_seen": 0,
"num_train_epochs": 3,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 758785239641088.0,
"train_batch_size": 1,
"trial_name": null,
"trial_params": null
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-2/training_args.bin
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-4/README.md
+207
View File
@@ -0,0 +1,207 @@
---
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:Qwen/Qwen2.5-1.5B-Instruct
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-4/adapter_config.json
+48
View File
@@ -0,0 +1,48 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "Qwen/Qwen2.5-1.5B-Instruct",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 64,
"lora_bias": false,
"lora_dropout": 0.05,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 32,
"rank_pattern": {},
"revision": null,
"target_modules": [
"up_proj",
"down_proj",
"k_proj",
"gate_proj",
"q_proj",
"v_proj",
"o_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-4/rng_state.pth
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-4/scheduler.pt
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-4/trainer_state.json
+62
View File
@@ -0,0 +1,62 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 2.0,
"eval_steps": 500,
"global_step": 4,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.6956521739130435,
"grad_norm": 0.45245930552482605,
"learning_rate": 2e-05,
"loss": 2.1483168601989746,
"step": 1
},
{
"epoch": 1.0,
"grad_norm": 0.5142671465873718,
"learning_rate": 1.6666666666666667e-05,
"loss": 2.2101056575775146,
"step": 2
},
{
"epoch": 1.6956521739130435,
"grad_norm": 0.39382970333099365,
"learning_rate": 1.3333333333333333e-05,
"loss": 2.0590713024139404,
"step": 3
},
{
"epoch": 2.0,
"grad_norm": 0.45731329917907715,
"learning_rate": 1e-05,
"loss": 2.369413137435913,
"step": 4
}
],
"logging_steps": 1,
"max_steps": 6,
"num_input_tokens_seen": 0,
"num_train_epochs": 3,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": false
},
"attributes": {}
}
},
"total_flos": 1517570479282176.0,
"train_batch_size": 1,
"trial_name": null,
"trial_params": null
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-4/training_args.bin
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-6/README.md
+207
View File
@@ -0,0 +1,207 @@
---
base_model: Qwen/Qwen2.5-1.5B-Instruct
library_name: peft
pipeline_tag: text-generation
tags:
- base_model:adapter:Qwen/Qwen2.5-1.5B-Instruct
- lora
- transformers
---
# Model Card for Model ID
<!-- Provide a quick summary of what the model is/does. -->
## Model Details
### Model Description
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [More Information Needed]
- **Funded by [optional]:** [More Information Needed]
- **Shared by [optional]:** [More Information Needed]
- **Model type:** [More Information Needed]
- **Language(s) (NLP):** [More Information Needed]
- **License:** [More Information Needed]
- **Finetuned from model [optional]:** [More Information Needed]
### Model Sources [optional]
<!-- Provide the basic links for the model. -->
- **Repository:** [More Information Needed]
- **Paper [optional]:** [More Information Needed]
- **Demo [optional]:** [More Information Needed]
## Uses
<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
### Direct Use
<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
[More Information Needed]
### Downstream Use [optional]
<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
[More Information Needed]
### Out-of-Scope Use
<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
[More Information Needed]
## Bias, Risks, and Limitations
<!-- This section is meant to convey both technical and sociotechnical limitations. -->
[More Information Needed]
### Recommendations
<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
## How to Get Started with the Model
Use the code below to get started with the model.
[More Information Needed]
## Training Details
### Training Data
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
[More Information Needed]
### Training Procedure
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
#### Preprocessing [optional]
[More Information Needed]
#### Training Hyperparameters
- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
#### Speeds, Sizes, Times [optional]
<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
[More Information Needed]
## Evaluation
<!-- This section describes the evaluation protocols and provides the results. -->
### Testing Data, Factors & Metrics
#### Testing Data
<!-- This should link to a Dataset Card if possible. -->
[More Information Needed]
#### Factors
<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
[More Information Needed]
#### Metrics
<!-- These are the evaluation metrics being used, ideally with a description of why. -->
[More Information Needed]
### Results
[More Information Needed]
#### Summary
## Model Examination [optional]
<!-- Relevant interpretability work for the model goes here -->
[More Information Needed]
## Environmental Impact
<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
- **Hardware Type:** [More Information Needed]
- **Hours used:** [More Information Needed]
- **Cloud Provider:** [More Information Needed]
- **Compute Region:** [More Information Needed]
- **Carbon Emitted:** [More Information Needed]
## Technical Specifications [optional]
### Model Architecture and Objective
[More Information Needed]
### Compute Infrastructure
[More Information Needed]
#### Hardware
[More Information Needed]
#### Software
[More Information Needed]
## Citation [optional]
<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
**BibTeX:**
[More Information Needed]
**APA:**
[More Information Needed]
## Glossary [optional]
<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
[More Information Needed]
## More Information [optional]
[More Information Needed]
## Model Card Authors [optional]
[More Information Needed]
## Model Card Contact
[More Information Needed]
### Framework versions
- PEFT 0.19.1
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-6/adapter_config.json
+48
View File
@@ -0,0 +1,48 @@
{
"alora_invocation_tokens": null,
"alpha_pattern": {},
"arrow_config": null,
"auto_mapping": null,
"base_model_name_or_path": "Qwen/Qwen2.5-1.5B-Instruct",
"bias": "none",
"corda_config": null,
"ensure_weight_tying": false,
"eva_config": null,
"exclude_modules": null,
"fan_in_fan_out": false,
"inference_mode": true,
"init_lora_weights": true,
"layer_replication": null,
"layers_pattern": null,
"layers_to_transform": null,
"loftq_config": {},
"lora_alpha": 64,
"lora_bias": false,
"lora_dropout": 0.05,
"lora_ga_config": null,
"megatron_config": null,
"megatron_core": "megatron.core",
"modules_to_save": null,
"peft_type": "LORA",
"peft_version": "0.19.1",
"qalora_group_size": 16,
"r": 32,
"rank_pattern": {},
"revision": null,
"target_modules": [
"up_proj",
"down_proj",
"k_proj",
"gate_proj",
"q_proj",
"v_proj",
"o_proj"
],
"target_parameters": null,
"task_type": "CAUSAL_LM",
"trainable_token_indices": null,
"use_bdlora": null,
"use_dora": false,
"use_qalora": false,
"use_rslora": false
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-6/rng_state.pth
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-6/scheduler.pt
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-6/trainer_state.json
+76
View File
@@ -0,0 +1,76 @@
{
"best_global_step": null,
"best_metric": null,
"best_model_checkpoint": null,
"epoch": 3.0,
"eval_steps": 500,
"global_step": 6,
"is_hyper_param_search": false,
"is_local_process_zero": true,
"is_world_process_zero": true,
"log_history": [
{
"epoch": 0.6956521739130435,
"grad_norm": 0.45245930552482605,
"learning_rate": 2e-05,
"loss": 2.1483168601989746,
"step": 1
},
{
"epoch": 1.0,
"grad_norm": 0.5142671465873718,
"learning_rate": 1.6666666666666667e-05,
"loss": 2.2101056575775146,
"step": 2
},
{
"epoch": 1.6956521739130435,
"grad_norm": 0.39382970333099365,
"learning_rate": 1.3333333333333333e-05,
"loss": 2.0590713024139404,
"step": 3
},
{
"epoch": 2.0,
"grad_norm": 0.45731329917907715,
"learning_rate": 1e-05,
"loss": 2.369413137435913,
"step": 4
},
{
"epoch": 2.6956521739130435,
"grad_norm": 0.3741995692253113,
"learning_rate": 6.666666666666667e-06,
"loss": 2.152299404144287,
"step": 5
},
{
"epoch": 3.0,
"grad_norm": 0.38987115025520325,
"learning_rate": 3.3333333333333333e-06,
"loss": 2.0988121032714844,
"step": 6
}
],
"logging_steps": 1,
"max_steps": 6,
"num_input_tokens_seen": 0,
"num_train_epochs": 3,
"save_steps": 500,
"stateful_callbacks": {
"TrainerControl": {
"args": {
"should_epoch_stop": false,
"should_evaluate": false,
"should_log": false,
"should_save": true,
"should_training_stop": true
},
"attributes": {}
}
},
"total_flos": 2276355718923264.0,
"train_batch_size": 1,
"trial_name": null,
"trial_params": null
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/checkpoint-6/training_args.bin
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/tokenizer_config.json
+30
View File
@@ -0,0 +1,30 @@
{
"add_prefix_space": false,
"backend": "tokenizers",
"bos_token": null,
"clean_up_tokenization_spaces": false,
"eos_token": "<|im_end|>",
"errors": "replace",
"extra_special_tokens": [
"<|im_start|>",
"<|im_end|>",
"<|object_ref_start|>",
"<|object_ref_end|>",
"<|box_start|>",
"<|box_end|>",
"<|quad_start|>",
"<|quad_end|>",
"<|vision_start|>",
"<|vision_end|>",
"<|vision_pad|>",
"<|image_pad|>",
"<|video_pad|>"
],
"is_local": false,
"local_files_only": false,
"model_max_length": 131072,
"pad_token": "<|endoftext|>",
"split_special_tokens": false,
"tokenizer_class": "Qwen2Tokenizer",
"unk_token": null
}
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/training_args.bin
BIN
View File
Binary file not shown.
chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1/training_manifest.json
+45
View File
@@ -0,0 +1,45 @@
{
"schema_version": 1,
"experiment": "8-9",
"stage": "student_sft",
"status": "complete",
"generated_at_utc": "2026-08-01T06:30:16.654563+00:00",
"book_git_commit": "88a5c5e86da9f9b0f97a46b4d501d715dc741e5a",
"training_data": {
"path": "/home/ubuntu/ai-agent-book/chapter8/cot-distillation/data/sft_cot_distill_aime_kimi_k3.jsonl",
"sha256": "0c6cab7cb8e0bd13671eda4e2bd0dc2530f7483f09ae42e946b87600a979968e",
"samples": 23
},
"base_model": "Qwen/Qwen2.5-1.5B-Instruct",
"output_dir": "/home/ubuntu/ai-agent-book/chapter8/cot-distillation/checkpoints/exp8-9-qwen25-1.5b-kimi-k3-20260801-v1",
"host": {
"platform": "Linux-6.8.0-111-generic-x86_64-with-glibc2.35",
"gpu_names": [
"NVIDIA RTX PRO 6000 Blackwell Workstation Edition"
],
"torch": "2.11.0+cu130"
},
"dependency_versions": {
"torch": "2.11.0",
"transformers": "5.14.1",
"accelerate": "1.14.0",
"peft": "0.19.1"
},
"training": {
"epochs": 3.0,
"learning_rate": 2e-05,
"max_length": 20000,
"batch_size": 1,
"gradient_accumulation": 16,
"lora_rank": 32,
"seed": 42,
"metrics": {
"train_runtime": 27.4522,
"train_samples_per_second": 2.513,
"train_steps_per_second": 0.219,
"total_flos": 2276355718923264.0,
"train_loss": 2.173003077507019,
"epoch": 3.0
}
}
}
chapter8/cot-distillation/data/raw_trajectories_aime.jsonl
+24
View File
File diff suppressed because one or more lines are too long
chapter8/cot-distillation/data/raw_trajectories_aime_kimi_k3.jsonl
+24
View File
File diff suppressed because one or more lines are too long

Some files were not shown because too many files have changed in this diff Show More