ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter8/continued-pretraining/README.md

@@ -0,0 +1,347 @@
+## English
+
+# Continued Pretraining: Teaching a Model a New Language (Korean Mistral)
+
+> This directory corresponds to Chapter 7, **Experiment 8-5 ★★: Continued Pretraining for Learning a New Language** of *Deep Understanding of AI Agents*.
+
+## Project Overview
+
+Using **Mistral 7B v0.3** as the base model (primarily pretrained on English, with virtually no understanding of Korean), we inject Korean language capability through **continued pretraining on Korean Wikipedia**, followed by **SFT on Korean instruction data**. The final model can both understand Korean and follow instructions in Korean.
+
+The core idea this experiment aims to demonstrate: **To make a model memorize a large amount of new domain knowledge (here, a new language), rely on continued pretraining, not SFT.** The model already possesses general language modeling ability from the pretraining phase; continued pretraining merely adapts it to a new data distribution, at a cost far lower than training from scratch.
+
+The entire process consists of two stages:
+
+1. **Continued Pretraining**: Unsupervised "predict the next token" training on Korean Wikipedia, allowing the model to learn Korean vocabulary and syntax.
+2. **Instruction Fine-Tuning (SFT)**: Training on Korean Alpaca instruction data to teach the model to "follow instructions in Korean."
+
+A key engineering challenge is **mitigating Catastrophic Forgetting**: learning a new language should not cause the model to forget its original English ability. The common approach discussed in the book uses mixed data (approximately 80% target language + 20% original language) to balance this; this implementation adopts a parameter-efficient scheme using **LoRA + training `embed_tokens`/`lm_head`** — only updating the adapters and word embeddings while keeping the base weights unchanged, thereby preserving English as much as possible while injecting Korean. Evaluation results (see below) show that English ability is largely retained.
+
+## Directory Structure
+
+```
+continued-pretraining/
+├── README.md                 # This document
+├── continued-pretrain.py     # Main training script: continued pretraining + SFT, produces two LoRA models
+├── evaluate_model.py         # Single model evaluation: generates samples on Korean/English tasks
+├── compare_models.py         # Three-stage comparison: base → continued pretraining → instruction fine-tuning side-by-side generation
+├── model_eval_results.md     # Full evaluation output and conclusions from actual run (RTX 4090)
+├── validation/               # Canonical report audit, blind-judge receipts, manifest, and validator
+├── README_EVALUATION.md      # Detailed usage instructions for evaluation scripts
+└── requirements.txt          # Dependency list
+```
+
+Running the training script produces two local directories (saving only LoRA adapters, not the full model):
+
+- `lora_model_pretrained/`: Model after continued pretraining, before SFT
+- `lora_model/`: Model after final instruction fine-tuning
+
+## System Requirements & Dependencies
+
+- **GPU**: Requires a CUDA-capable NVIDIA GPU. By default, Mistral-7B is loaded in 4-bit quantization, allowing training on consumer-grade GPUs with approximately 24GB VRAM (e.g., RTX 4090). The results in `model_eval_results.md` were produced on an RTX 4090.
+- **Framework**: [Unsloth](https://github.com/unslothai/unsloth) (efficient LoRA training), PyTorch, Transformers, Datasets, bitsandbytes.
+- **Optional**: wandb (experiment tracking; script defaults to `report_to="wandb"`).
+
+```bash
+# From the repository root: use the shared Chapter 7 environment plus Unsloth
+uv sync --locked --python 3.12 --extra ch7 --extra unsloth
+
+# Activate it before changing directories:
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .\.venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+# pip fallback when uv is not installed:
+# python -m pip install -e ".[ch7,unsloth]"
+
+cd chapter8/continued-pretraining
+
+# Single-project compatibility path, still supported for exact legacy parity
+# (including the original Unsloth Git install used by this project):
+# python -m pip install -r requirements.txt
+```
+
+> Note: Unsloth depends on a GPU with a compatible CUDA/PyTorch version and cannot be used for training or inference in a pure CPU environment. The `--help` for each script uses lazy imports, so parameter descriptions can be viewed on machines without a GPU.
+
+## Quick Start
+
+### 1. Training (Continued Pretraining + SFT)
+
+Run both stages with default hyperparameters in one command (using a 5% subset of Korean Wikipedia for continued pretraining, followed by SFT on Korean Alpaca):
+
+```bash
+python continued-pretrain.py
+```
+
+The script will sequentially: load the base model → print a baseline test → perform continued pretraining on Korean Wikipedia → save `lora_model_pretrained/` → perform SFT on Korean instructions → save `lora_model/`.
+
+Common parameters (defaults match the script's original hardcoded values; changes will deviate from the original experiment):
+
+```bash
+python continued-pretrain.py \
+    --base_model unsloth/mistral-7b-v0.3 \
+    --wiki_config 20231101.ko \
+    --wiki_train_size 0.05 \
+    --alpaca_dataset FreedomIntelligence/alpaca-gpt4-korean \
+    --lora_rank 128 \
+    --max_seq_len 2048 \
+    --pretrain_epochs 1 \
+    --sft_epochs 2 \
+    --pretrained_save_dir lora_model_pretrained \
+    --final_save_dir lora_model
+```
+
+- For a quick smoke test, use `--pretrain_max_steps 20 --sft_max_steps 20` to run only a few steps.
+- To switch to a different language: replace `--wiki_config` with the corresponding Wikipedia snapshot (e.g., `20231101.ja` for Japanese) and `--alpaca_dataset` with the corresponding instruction dataset.
+- See `python continued-pretrain.py --help` for the full list of parameters.
+
+### 2. Evaluating a Single Model
+
+```bash
+# Evaluate the final fine-tuned model (default loads lora_model/)
+python evaluate_model.py
+
+# Evaluate the model after continued pretraining, before SFT
+python evaluate_model.py --pretrained
+
+# Generate longer outputs using sampling
+python evaluate_model.py --max_new_tokens 300 --use_sampling --temperature 0.7
+```
+
+See [`README_EVALUATION.md`](./README_EVALUATION.md) for more usage details.
+
+### 3. Three-Stage Side-by-Side Comparison
+
+Load the **base model / continued pretraining model / instruction fine-tuning model** simultaneously and generate side-by-side outputs on the same set of Korean and English prompts, visually demonstrating the improvement in Korean ability and the retention of English ability:
+
+```bash
+python compare_models.py
+```
+
+```bash
+# Specify model directories and generation parameters
+python compare_models.py \
+    --pretrained_path lora_model_pretrained \
+    --finetuned_path lora_model \
+    --max_new_tokens 150 \
+    --temperature 0.3
+```
+
+## Experimental Results
+
+The full terminal output from the historical RTX 4090 run is retained in [`model_eval_results.md`](./model_eval_results.md). The canonical evidence package is [`validation/runs/exp8-5-training-report-20260731-v1/`](validation/runs/exp8-5-training-report-20260731-v1/), and [`validation/latest.json`](validation/latest.json) binds its manifest.
+
+The audit extracted all **5 prompts × 3 stages = 15 outputs** and sent five deterministic stage-blind comparison tasks to the independent ARK `doubao-seed-1-6-250615` judge. All five raw request/responses, unique response IDs, usage, and latency are retained. The 0–5 mean scores were:
+
+| Stage | Korean | English |
+| --- | ---: | ---: |
+| Base Mistral | 1.6667 | 5.0000 |
+| After Korean continued pretraining | 1.3333 | 3.1667 |
+| After Korean instruction SFT | 3.4444 | 4.1667 |
+
+The final stage improved the Korean mean by **+1.7777** over baseline. Its English mean fell by **0.8333**, within the audit's declared 1.0-point retention tolerance. Continued pretraining alone did not improve this small retained prompt set; the final SFT stage produced the observed Korean gain. The kimchi answer remained materially false: it described boiling vegetables and soaking them in a soy-sauce-based sauce. That limitation is an accepted negative result, not hidden by the aggregate score.
+
+The historical run did not retain adapter hashes, exact resolved upstream commits, or its generation seed. For future reproduction, the audit freezes immutable current revisions for the base model and both datasets in [`reproduction_contract.json`](validation/runs/exp8-5-training-report-20260731-v1/reproduction_contract.json). Those pins are explicitly **not claimed to be the historical revisions**.
+
+Training checkpoints/adapters are intentionally local and are not distributed with the book. They are not acceptance artifacts; the accepted artifact is the reproducible, evidence-backed report. Validate it without a GPU or provider call:
+
+```bash
+python chapter8/continued-pretraining/validation/validate_evidence.py
+python -m pytest chapter8/continued-pretraining/validation/test_report_audit.py -q
+```
+
+To create a new independent audit from the retained report, set `ARK_API_KEY` and use a new run ID:
+
+```bash
+python chapter8/continued-pretraining/validation/run_report_audit.py \
+  --run-id exp8-5-training-report-YYYYMMDD-vN
+```
+
+The evidence-backed conclusions are:
+
+- **The full two-stage path improved Korean in this retained comparison**: the final SFT stage scored substantially above the baseline, while the continued-pretrained intermediate stage did not.
+- **English remained usable but measurably regressed**: the final stage stayed within the declared tolerance; the intermediate stage regressed much more.
+- **Fluency is not factual reliability**: the fluent final kimchi answer contains serious preparation and ingredient errors.
+
+## References
+
+- Unsloth documentation: https://docs.unsloth.ai
+- Base model: [unsloth/mistral-7b-v0.3](https://huggingface.co/unsloth/mistral-7b-v0.3)
+- Continued pretraining corpus: [wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia) (`20231101.ko`)
+- Instruction fine-tuning corpus: [FreedomIntelligence/alpaca-gpt4-korean](https://huggingface.co/datasets/FreedomIntelligence/alpaca-gpt4-korean)
+
+---
+
+## 中文
+
+# 继续预训练：让模型学会一门新语言（韩语 Mistral）
+
+> 本目录对应《深入理解 AI Agent》第 7 章 **实验 8-5 ★★：继续预训练学习新语言**。
+
+## 项目简介
+
+以 **Mistral 7B v0.3** 为基础模型（主要用英语预训练，对韩语几乎没有理解能力），通过**韩语维基百科继续预训练**注入韩语能力，再用**韩语指令数据做 SFT**，最终得到一个既能理解韩语、又能用韩语遵循指令的模型。
+
+本实验想说明的核心观点：**要让模型记住大量新领域知识（这里是一门新语言），靠的是继续预训练，而不是 SFT。** 模型在预训练阶段已经具备通用的语言建模能力，继续预训练只是让它适应新的数据分布，成本远低于从头训练。
+
+整个流程分两个阶段：
+
+1. **继续预训练（Continued Pretraining）**：在韩语维基百科上做无监督的“预测下一个词”训练，让模型学会韩语的词汇与句法。
+2. **指令微调（SFT）**：在韩语 Alpaca 指令数据上训练，让模型学会“用韩语遵循指令”。
+
+一个关键工程点是**缓解灾难性遗忘（Catastrophic Forgetting）**：学了新语言不能把原来的英语能力忘掉。书中讨论的通用做法是用混合数据（约 80% 目标语言 + 20% 原语言）来平衡；本实现则采用 **LoRA + 训练 `embed_tokens`/`lm_head`** 的参数高效方案——只更新适配器与词嵌入，基础权重保持不变，从而在注入韩语的同时尽量保留英语。评测结果（见下文）显示英语能力基本得到保留。
+
+## 目录结构
+
+```
+continued-pretraining/
+├── README.md                 # 本文档
+├── continued-pretrain.py     # 训练主脚本：继续预训练 + SFT，产出两个 LoRA 模型
+├── evaluate_model.py         # 单模型评测：在韩英任务上生成样例
+├── compare_models.py         # 三阶段对比：基础 → 继续预训练 → 指令微调 并排生成
+├── model_eval_results.md     # 真实运行的完整评测输出与结论（RTX 4090）
+├── validation/               # 规范报告审计、盲评回执、manifest 与验证器
+├── README_EVALUATION.md      # 评测脚本的详细用法说明
+└── requirements.txt          # 依赖清单
+```
+
+训练脚本运行后会产出两个本地目录（仅保存 LoRA 适配器，不含完整模型）：
+
+- `lora_model_pretrained/`：继续预训练之后、SFT 之前的模型
+- `lora_model/`：最终指令微调之后的模型
+
+## 系统要求与依赖
+
+- **GPU**：需要支持 CUDA 的 NVIDIA GPU。默认以 4bit 量化加载 Mistral-7B，可在约 24GB 显存的消费级显卡（如 RTX 4090）上完成训练，`model_eval_results.md` 中的结果即在 RTX 4090 上产出。
+- **框架**：[Unsloth](https://github.com/unslothai/unsloth)（高效 LoRA 训练）、PyTorch、Transformers、Datasets、bitsandbytes。
+- **可选**：wandb（实验跟踪，脚本默认 `report_to="wandb"`）。
+
+```bash
+# 在仓库根目录使用统一的第 7 章环境，并显式加入 Unsloth
+uv sync --locked --python 3.12 --extra ch7 --extra unsloth
+
+# 切换目录前先激活环境：
+# macOS/Linux：
+source .venv/bin/activate
+# Windows PowerShell：.\.venv\Scripts\Activate.ps1
+# Windows cmd：.venv\Scripts\activate.bat
+
+# 未安装 uv 时可用 pip 兜底：
+# python -m pip install -e ".[ch7,unsloth]"
+
+cd chapter8/continued-pretraining
+
+# 迁移期间仍支持单项目兼容路径，用于完全复现旧版依赖
+#（包括本项目原有的 Unsloth Git 安装方式）：
+# python -m pip install -r requirements.txt
+```
+
+> 注意：Unsloth 依赖 GPU 与匹配的 CUDA/PyTorch 版本，无法在纯 CPU 环境下训练或推理。各脚本的 `--help` 已做延迟导入，可在没有 GPU 的机器上直接查看参数说明。
+
+## 快速开始
+
+### 1. 训练（继续预训练 + SFT）
+
+用默认超参数一键完成两个阶段（韩语维基百科 5% 子集做继续预训练，随后用韩语 Alpaca 做 SFT）：
+
+```bash
+python continued-pretrain.py
+```
+
+脚本会依次：加载基础模型 → 打印基线测试 → 韩语维基继续预训练 → 保存 `lora_model_pretrained/` → 韩语指令 SFT → 保存 `lora_model/`。
+
+常用参数（默认值与脚本原始硬编码一致，改动才会偏离原实验）：
+
+```bash
+python continued-pretrain.py \
+    --base_model unsloth/mistral-7b-v0.3 \
+    --wiki_config 20231101.ko \
+    --wiki_train_size 0.05 \
+    --alpaca_dataset FreedomIntelligence/alpaca-gpt4-korean \
+    --lora_rank 128 \
+    --max_seq_len 2048 \
+    --pretrain_epochs 1 \
+    --sft_epochs 2 \
+    --pretrained_save_dir lora_model_pretrained \
+    --final_save_dir lora_model
+```
+
+- 想快速冒烟测试，可用 `--pretrain_max_steps 20 --sft_max_steps 20` 只跑很少的步数。
+- 想换一门语言：把 `--wiki_config` 换成对应维基快照（如 `20231101.ja` 日语）、`--alpaca_dataset` 换成对应语言的指令集即可。
+- 完整参数见 `python continued-pretrain.py --help`。
+
+### 2. 评测单个模型
+
+```bash
+# 评测最终微调模型（默认加载 lora_model/）
+python evaluate_model.py
+
+# 评测继续预训练后、SFT 前的模型
+python evaluate_model.py --pretrained
+
+# 生成更长、使用采样
+python evaluate_model.py --max_new_tokens 300 --use_sampling --temperature 0.7
+```
+
+更多用法详见 [`README_EVALUATION.md`](./README_EVALUATION.md)。
+
+### 3. 三阶段并排对比
+
+同时加载**基础模型 / 继续预训练模型 / 指令微调模型**，在同一组中韩英提示上并排生成，直观展示韩语能力的提升与英语能力的保留：
+
+```bash
+python compare_models.py
+```
+
+```bash
+# 指定模型目录与生成参数
+python compare_models.py \
+    --pretrained_path lora_model_pretrained \
+    --finetuned_path lora_model \
+    --max_new_tokens 150 \
+    --temperature 0.3
+```
+
+## 实验结果
+
+历史 RTX 4090 运行的完整终端输出保存在 [`model_eval_results.md`](./model_eval_results.md)。规范证据包位于 [`validation/runs/exp8-5-training-report-20260731-v1/`](validation/runs/exp8-5-training-report-20260731-v1/)，[`validation/latest.json`](validation/latest.json) 绑定其 manifest。
+
+审计从原始报告提取了 **5 个提示 × 3 个阶段 = 15 个输出**，并向独立 ARK `doubao-seed-1-6-250615` 裁判提交了五次确定性乱序、阶段匿名的对比。五份原始请求/响应、唯一 response ID、usage 与延迟均已保留。0–5 分均值如下：
+
+| 阶段 | 韩语 | 英语 |
+| --- | ---: | ---: |
+| 基础 Mistral | 1.6667 | 5.0000 |
+| 韩语继续预训练后 | 1.3333 | 3.1667 |
+| 韩语指令 SFT 后 | 3.4444 | 4.1667 |
+
+最终阶段相对基线的韩语均值提升 **+1.7777**；英语均值下降 **0.8333**，仍在预先声明的 1.0 分保留容差内。仅继续预训练的中间阶段在这组小规模保留提示上没有提升，观察到的韩语增益来自完整两阶段流程后的最终 SFT 模型。最终模型的泡菜回答仍有严重事实错误：它错误地描述了煮蔬菜和以酱油为基础的浸泡汁。这个负结果被明确保留，而没有被总分掩盖。
+
+历史运行没有保留 adapter hash、当时解析到的上游 commit 或生成随机种子。为了将来复现，[`reproduction_contract.json`](validation/runs/exp8-5-training-report-20260731-v1/reproduction_contract.json) 固定了基础模型和两个数据集的当前不可变 revision；这些 revision 明确**不声称是历史运行所用版本**。
+
+训练 checkpoint/adapter 按本书策略仅保存在本地，不随书分发，也不是验收产物；验收产物是可复现、证据充分的训练报告。无需 GPU 或 API 调用即可验证：
+
+```bash
+python chapter8/continued-pretraining/validation/validate_evidence.py
+python -m pytest chapter8/continued-pretraining/validation/test_report_audit.py -q
+```
+
+如需从保留报告创建新的独立审计，设置 `ARK_API_KEY` 并使用新的 run ID：
+
+```bash
+python chapter8/continued-pretraining/validation/run_report_audit.py \
+  --run-id exp8-5-training-report-YYYYMMDD-vN
+```
+
+有证据支持的结论如下：
+
+- **完整两阶段流程在本次保留对比中提升了韩语**：最终 SFT 阶段显著高于基线，但继续预训练的中间阶段没有提升。
+- **英语仍可用，但出现可测量的退化**：最终阶段仍在声明容差内；中间阶段退化更明显。
+- **流畅不等于事实可靠**：最终泡菜回答虽然更流畅，却包含严重的制作方法与配料错误。
+
+## 参考资料
+
+- Unsloth 文档：https://docs.unsloth.ai
+- 基础模型：[unsloth/mistral-7b-v0.3](https://huggingface.co/unsloth/mistral-7b-v0.3)
+- 继续预训练语料：[wikimedia/wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)（`20231101.ko`）
+- 指令微调语料：[FreedomIntelligence/alpaca-gpt4-korean](https://huggingface.co/datasets/FreedomIntelligence/alpaca-gpt4-korean)

chapter8/continued-pretraining/README_EVALUATION.md

@@ -0,0 +1,247 @@
+# Korean Mistral Model Evaluation Guide
+
+This guide explains how to use the evaluation script to test your trained Korean Mistral models.
+
+## Overview
+
+After running `continued-pretrain.py`, you'll have two saved models:
+- `lora_model_pretrained/` - Model after Korean pretraining (before instruction finetuning)
+- `lora_model/` - Final model after instruction finetuning
+
+## Quick Start
+
+### Basic Evaluation (Final Finetuned Model)
+
+```bash
+python evaluate_model.py
+```
+
+This will:
+- Load the final finetuned model from `lora_model/`
+- Run 6 test cases (Korean + English, Wikipedia + Instructions)
+- Use default parameters (max_new_tokens=150)
+
+### Evaluate Pretrained Model (Before SFT)
+
+```bash
+python evaluate_model.py --pretrained
+```
+
+This loads the model after Korean pretraining but before instruction finetuning.
+
+## Command Line Options
+
+### Model Selection
+
+```bash
+# Evaluate the pretrained model
+python evaluate_model.py --pretrained
+
+# Evaluate a custom model path
+python evaluate_model.py --model_path path/to/your/model
+
+# Load in full precision (more memory, higher quality)
+python evaluate_model.py --load_in_4bit False
+```
+
+### Generation Parameters
+
+```bash
+# Generate more tokens
+python evaluate_model.py --max_new_tokens 300
+
+# Use sampling for more creative outputs
+python evaluate_model.py --use_sampling --temperature 0.8 --top_p 0.95
+```
+
+### All Available Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--model_path` | `lora_model` | Path to saved LoRA model |
+| `--pretrained` | `False` | Load pretrained model (before SFT) |
+| `--max_seq_length` | `2048` | Maximum sequence length |
+| `--load_in_4bit` | `True` | Use 4-bit quantization |
+| `--max_new_tokens` | `150` | Maximum tokens to generate |
+| `--use_sampling` | `False` | Enable sampling (vs greedy) |
+| `--temperature` | `0.7` | Sampling temperature (creativity) |
+| `--top_p` | `0.9` | Top-p nucleus sampling |
+
+## Example Use Cases
+
+### Compare Models Side-by-Side
+
+```bash
+# First, test the pretrained model
+python evaluate_model.py --pretrained > results_pretrained.txt
+
+# Then, test the finetuned model
+python evaluate_model.py > results_finetuned.txt
+
+# Compare the outputs
+diff results_pretrained.txt results_finetuned.txt
+```
+
+### Creative vs Deterministic Generation
+
+```bash
+# Deterministic (greedy decoding) - same output every time
+python evaluate_model.py
+
+# Creative (sampling) - different output each time
+python evaluate_model.py --use_sampling --temperature 0.7
+
+# Very creative (higher temperature)
+python evaluate_model.py --use_sampling --temperature 1.0
+
+# More focused (lower temperature)
+python evaluate_model.py --use_sampling --temperature 0.3
+```
+
+### Long-Form Generation
+
+```bash
+# Generate longer responses
+python evaluate_model.py --max_new_tokens 500
+```
+
+## Test Cases
+
+### Evaluation Script (evaluate_model.py)
+Runs 6 test cases on a single model:
+
+1. **Korean Wikipedia Article (Artificial Intelligence)** - Tests encyclopedic writing in Korean
+2. **English Wikipedia Article (Artificial Intelligence)** - Ensures English preservation
+3. **Korean Instruction (Explain Kimchi)** - Tests instruction-following for cultural topics
+4. **English Instruction (Explain Thanksgiving Turkey)** - Tests English instruction-following
+5. **Korean Instruction (Introduce Seoul)** - Tests factual knowledge in Korean
+6. **Korean Instruction (Explain K-pop)** - Tests modern cultural knowledge
+
+### Comparison Script (compare_models.py)
+Runs 5 test cases across 3 models (15 total outputs):
+
+1. **Korean Wikipedia - AI** - Shows Korean capability progression
+2. **English Wikipedia - AI** - Validates English preservation (encyclopedic writing)
+3. **Korean Instruction - Kimchi** - Shows instruction-following improvement
+4. **Korean Instruction - Seoul** - Tests factual accuracy improvement
+5. **English Instruction - Thanksgiving** - Validates English preservation (instruction-following)
+
+The comparison script includes both English Wikipedia AND English Instruction tests to comprehensively validate that English capabilities remain strong throughout all training stages.
+
+## Understanding the Output
+
+### Color Coding
+- 🔵 **Blue**: Loading and setup information
+- 🟡 **Yellow**: Parameters and configuration
+- 🟢 **Green**: Successful operations and output
+- 🔴 **Red**: Errors
+- 🔵 **Cyan**: Prompts and tips
+
+### Evaluation Metrics (Manual)
+
+When evaluating outputs, consider:
+
+1. **Fluency**: Is the Korean grammatically correct?
+2. **Factual Accuracy**: Are the facts correct?
+3. **Instruction Following**: Does it answer the question?
+4. **Coherence**: Does it make logical sense?
+5. **Cultural Appropriateness**: Is cultural information accurate?
+
+## Troubleshooting
+
+### "Model path does not exist"
+Make sure you've run `continued-pretrain.py` first to train and save the models.
+
+### Out of Memory
+Try:
+```bash
+# Use 4-bit quantization
+python evaluate_model.py --load_in_4bit
+
+# Reduce max sequence length
+python evaluate_model.py --max_seq_length 1024
+
+# Generate fewer tokens
+python evaluate_model.py --max_new_tokens 100
+```
+
+### Outputs Too Short
+Increase max tokens:
+```bash
+python evaluate_model.py --max_new_tokens 300
+```
+
+### Want Different Outputs Each Time
+Enable sampling:
+```bash
+python evaluate_model.py --use_sampling
+```
+
+## Tips for Best Results
+
+1. **Start with defaults**: Run with no arguments first
+2. **Compare stages**: Test both `--pretrained` and final model
+3. **Use sampling for variety**: Add `--use_sampling` for creative outputs
+4. **Monitor GPU memory**: Check the memory stats in output
+
+## Expected Performance
+
+### Baseline Model (No Training)
+- ❌ Korean: Poor, repetitive, often nonsensical
+- ✅ English: Good, coherent, accurate
+
+### Pretrained Model (After Korean Training)
+- ⚠️ Korean: Improved fluency, better vocabulary
+- ✅ English: Maintained quality
+- ⚠️ Instructions: Better than baseline, but not perfect
+
+### Finetuned Model (After SFT)
+- ✅ Korean: Fluent, accurate, follows instructions
+- ✅ English: Maintained quality
+- ✅ Instructions: Good instruction-following in both languages
+
+## Advanced Usage
+
+### Batch Testing Multiple Configurations
+
+Create a shell script:
+
+```bash
+#!/bin/bash
+# test_configs.sh
+
+echo "Testing different temperatures..."
+
+for temp in 0.3 0.7 1.0; do
+    echo "=== Testing temperature=$temp ==="
+    python evaluate_model.py --use_sampling --temperature $temp \
+        --max_new_tokens 150 > results_temp_${temp}.txt
+done
+
+echo "Testing different token lengths..."
+
+for tokens in 100 200 300; do
+    echo "=== Testing max_new_tokens=$tokens ==="
+    python evaluate_model.py --max_new_tokens $tokens \
+        > results_tokens_${tokens}.txt
+done
+```
+
+### Custom Test Prompts
+
+Modify the `run_evaluation()` function in `evaluate_model.py` to add your own test cases.
+
+## References
+
+- Main training script: `continued-pretrain.py`
+- Unsloth documentation: https://docs.unsloth.ai
+- Generation parameters: https://huggingface.co/docs/transformers/main_classes/text_generation
+
+## Support
+
+If you encounter issues:
+1. Check that training completed successfully
+2. Verify model files exist in `lora_model/` or `lora_model_pretrained/`
+3. Ensure you have sufficient GPU memory
+4. Try reducing `--max_seq_length` or `--max_new_tokens`
+

chapter8/continued-pretraining/compare_models.py

@@ -0,0 +1,296 @@
+# -*- coding: utf-8 -*-
+"""
+Compare baseline → pretrained → finetuned Korean Mistral models (3-way comparison)
+Shows progression from original model to final Korean-capable model
+"""
+
+import argparse
+
+# 说明：unsloth / torch 等重型依赖在函数内按需导入，
+# 这样 `python compare_models.py --help` 无需 GPU 环境即可查看参数。
+
+# ANSI color codes for colored output
+class Colors:
+    HEADER = '\033[95m'
+    BLUE = '\033[94m'
+    CYAN = '\033[96m'
+    GREEN = '\033[92m'
+    YELLOW = '\033[93m'
+    RED = '\033[91m'
+    ENDC = '\033[0m'
+    BOLD = '\033[1m'
+    UNDERLINE = '\033[4m'
+
+def print_section(title, color=Colors.CYAN):
+    """Print a colored section header"""
+    print(f"\n{color}{Colors.BOLD}{'='*80}")
+    print(f"{title}")
+    print(f"{'='*80}{Colors.ENDC}\n")
+
+def load_baseline_model(base_model="unsloth/mistral-7b-v0.3", max_seq_length=2048):
+    """Load the original Mistral model (before any training)"""
+    from unsloth import FastLanguageModel
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name=base_model,
+        max_seq_length=max_seq_length,
+        dtype=None,
+        load_in_4bit=True,
+    )
+    FastLanguageModel.for_inference(model)
+    return model, tokenizer
+
+def load_model(model_path, max_seq_length=2048):
+    """Load a trained LoRA model"""
+    from unsloth import FastLanguageModel
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name=model_path,
+        max_seq_length=max_seq_length,
+        dtype=None,
+        load_in_4bit=True,
+    )
+    FastLanguageModel.for_inference(model)
+    return model, tokenizer
+
+def generate_text(model, tokenizer, prompt, max_new_tokens=150, temperature=0.3):
+    """Generate text without streaming"""
+    inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
+    outputs = model.generate(
+        **inputs,
+        max_new_tokens=max_new_tokens,
+        use_cache=True,
+        do_sample=True,
+        temperature=temperature,
+        pad_token_id=tokenizer.eos_token_id,
+    )
+    generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
+    # Remove the prompt from the output
+    response = generated_text[len(prompt):].strip()
+    return response
+
+def compare_on_prompt(baseline_model, baseline_tokenizer,
+                     pretrained_model, pretrained_tokenizer, 
+                     finetuned_model, finetuned_tokenizer,
+                     prompt, test_name, prompt_translation=None,
+                     max_new_tokens=150, temperature=0.3):
+    """Compare three models on the same prompt"""
+    
+    print(f"\n{Colors.BOLD}{'='*80}")
+    print(f"{test_name}")
+    print(f"{'='*80}{Colors.ENDC}")
+    
+    if prompt_translation:
+        print(f"{Colors.CYAN}Prompt (Translation): {prompt_translation}{Colors.ENDC}\n")
+    
+    print(f"{Colors.YELLOW}Generating from BASELINE model (original Mistral)...{Colors.ENDC}")
+    baseline_output = generate_text(
+        baseline_model, baseline_tokenizer, prompt, 
+        max_new_tokens, temperature
+    )
+    
+    print(f"{Colors.YELLOW}Generating from PRETRAINED model (after Korean training)...{Colors.ENDC}")
+    pretrained_output = generate_text(
+        pretrained_model, pretrained_tokenizer, prompt, 
+        max_new_tokens, temperature
+    )
+    
+    print(f"{Colors.YELLOW}Generating from FINETUNED model (after instruction tuning)...{Colors.ENDC}")
+    finetuned_output = generate_text(
+        finetuned_model, finetuned_tokenizer, prompt,
+        max_new_tokens, temperature
+    )
+    
+    # Display all three outputs
+    print(f"\n{Colors.RED}┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐{Colors.ENDC}")
+    print(f"{Colors.RED}│{Colors.ENDC}")
+    for line in baseline_output.split('\n'):
+        print(f"{Colors.RED}│{Colors.ENDC} {line}")
+    print(f"{Colors.RED}│{Colors.ENDC}")
+    print(f"{Colors.RED}└────────────────────────────────────────────────────────────────────┘{Colors.ENDC}\n")
+    
+    print(f"{Colors.GREEN}┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐{Colors.ENDC}")
+    print(f"{Colors.GREEN}│{Colors.ENDC}")
+    for line in pretrained_output.split('\n'):
+        print(f"{Colors.GREEN}│{Colors.ENDC} {line}")
+    print(f"{Colors.GREEN}│{Colors.ENDC}")
+    print(f"{Colors.GREEN}└────────────────────────────────────────────────────────────────────┘{Colors.ENDC}\n")
+    
+    print(f"{Colors.CYAN}┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐{Colors.ENDC}")
+    print(f"{Colors.CYAN}│{Colors.ENDC}")
+    for line in finetuned_output.split('\n'):
+        print(f"{Colors.CYAN}│{Colors.ENDC} {line}")
+    print(f"{Colors.CYAN}│{Colors.ENDC}")
+    print(f"{Colors.CYAN}└────────────────────────────────────────────────────────────────────┘{Colors.ENDC}\n")
+
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description="对比韩语 Mistral 的三个阶段模型：基础模型 → 继续预训练 → 指令微调。"
+                    "在同一批中韩英提示上并排生成，直观展示韩语能力的提升与英语能力的保留。",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+    parser.add_argument("--base_model", type=str, default="unsloth/mistral-7b-v0.3",
+                        help="基础（未训练）模型名称")
+    parser.add_argument("--pretrained_path", type=str, default="lora_model_pretrained",
+                        help="继续预训练后保存的 LoRA 模型目录")
+    parser.add_argument("--finetuned_path", type=str, default="lora_model",
+                        help="指令微调后保存的最终 LoRA 模型目录")
+    parser.add_argument("--max_seq_length", type=int, default=2048,
+                        help="最大序列长度")
+    parser.add_argument("--max_new_tokens", type=int, default=150,
+                        help="每次生成的最大 token 数")
+    parser.add_argument("--temperature", type=float, default=0.3,
+                        help="采样温度（越低越确定）")
+    return parser.parse_args()
+
+
+def main():
+    args = parse_args()
+
+    print_section("🔬 KOREAN MISTRAL 3-WAY MODEL COMPARISON", Colors.HEADER)
+    print(f"{Colors.YELLOW}This script compares three model stages:{Colors.ENDC}")
+    print(f"  1. {Colors.RED}Baseline{Colors.ENDC} - Original Mistral (no Korean training)")
+    print(f"  2. {Colors.GREEN}Pretrained{Colors.ENDC} - After Korean Wikipedia training")
+    print(f"  3. {Colors.CYAN}Finetuned{Colors.ENDC} - After instruction tuning")
+    print(f"\n{Colors.CYAN}Generation settings: temperature={args.temperature}, do_sample=True (no repetition_penalty){Colors.ENDC}\n")
+    
+    # Load all three models
+    print_section("📥 LOADING MODELS", Colors.BLUE)
+    
+    print(f"{Colors.YELLOW}Loading baseline model (original Mistral v0.3)...{Colors.ENDC}")
+    baseline_model, baseline_tokenizer = load_baseline_model(args.base_model, args.max_seq_length)
+    print(f"{Colors.GREEN}✓ Baseline model loaded{Colors.ENDC}")
+    
+    print(f"\n{Colors.YELLOW}Loading pretrained model (after Korean pretraining)...{Colors.ENDC}")
+    pretrained_model, pretrained_tokenizer = load_model(args.pretrained_path, args.max_seq_length)
+    print(f"{Colors.GREEN}✓ Pretrained model loaded{Colors.ENDC}")
+    
+    print(f"\n{Colors.YELLOW}Loading finetuned model (after instruction tuning)...{Colors.ENDC}")
+    finetuned_model, finetuned_tokenizer = load_model(args.finetuned_path, args.max_seq_length)
+    print(f"{Colors.GREEN}✓ Finetuned model loaded{Colors.ENDC}")
+    
+    # Define prompts
+    wikipedia_prompt_korean = """위키피디아 기사
+### 제목: {}
+
+### 기사:
+{}"""
+
+    wikipedia_prompt_english = """Wikipedia Article
+### Title: {}
+
+### Article:
+{}"""
+
+    alpaca_prompt_korean = """다음은 작업을 설명하는 명령입니다. 요청을 적절하게 완료하는 응답을 작성하세요.
+
+### 지침:
+{}
+
+### 응답:
+{}"""
+
+    alpaca_prompt_english = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+### Instruction:
+{}
+
+### Response:
+{}"""
+    
+    print_section("🧪 RUNNING 3-WAY COMPARISONS", Colors.CYAN)
+    
+    # Test 1: Korean Wikipedia
+    compare_on_prompt(
+        baseline_model, baseline_tokenizer,
+        pretrained_model, pretrained_tokenizer,
+        finetuned_model, finetuned_tokenizer,
+        wikipedia_prompt_korean.format("인공지능", ""),
+        "Test 1: Korean Wikipedia - Artificial Intelligence (인공지능)",
+        "Wikipedia Article / Title: Artificial Intelligence / Article:",
+        max_new_tokens=args.max_new_tokens, temperature=args.temperature
+    )
+    
+    # Test 2: English Wikipedia - Preservation Check
+    compare_on_prompt(
+        baseline_model, baseline_tokenizer,
+        pretrained_model, pretrained_tokenizer,
+        finetuned_model, finetuned_tokenizer,
+        wikipedia_prompt_english.format("Artificial Intelligence", ""),
+        "Test 2: English Wikipedia - Artificial Intelligence (Preservation Check)",
+        None,
+        max_new_tokens=args.max_new_tokens, temperature=args.temperature
+    )
+    
+    # Test 3: Korean Instruction - Kimchi
+    compare_on_prompt(
+        baseline_model, baseline_tokenizer,
+        pretrained_model, pretrained_tokenizer,
+        finetuned_model, finetuned_tokenizer,
+        alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", ""),
+        "Test 3: Korean Instruction - Explain Kimchi",
+        "Instruction: Explain about kimchi, a traditional Korean food. / Response:",
+        max_new_tokens=args.max_new_tokens, temperature=args.temperature
+    )
+    
+    # Test 4: Korean Instruction - Seoul
+    compare_on_prompt(
+        baseline_model, baseline_tokenizer,
+        pretrained_model, pretrained_tokenizer,
+        finetuned_model, finetuned_tokenizer,
+        alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", ""),
+        "Test 4: Korean Instruction - Introduce Seoul",
+        "Instruction: Briefly introduce Seoul, the capital of South Korea. / Response:",
+        max_new_tokens=args.max_new_tokens, temperature=args.temperature
+    )
+    
+    # Test 5: English Instruction - Preservation Check
+    compare_on_prompt(
+        baseline_model, baseline_tokenizer,
+        pretrained_model, pretrained_tokenizer,
+        finetuned_model, finetuned_tokenizer,
+        alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", ""),
+        "Test 5: English Instruction - Thanksgiving Turkey (Preservation Check)",
+        None,
+        max_new_tokens=args.max_new_tokens, temperature=args.temperature
+    )
+    
+    print_section("📊 COMPARISON COMPLETE", Colors.GREEN)
+    
+    print(f"{Colors.CYAN}{'='*80}")
+    print(f"💡 What to Look For:")
+    print(f"{'='*80}{Colors.ENDC}")
+    
+    print(f"\n{Colors.RED}Baseline Model (Red boxes - Original Mistral):{Colors.ENDC}")
+    print(f"  • Korean: Should be POOR - repetitive, nonsensical")
+    print(f"  • English: Should be GOOD - this is the starting point")
+    print(f"  • Shows what model knows BEFORE any Korean training")
+    
+    print(f"\n{Colors.GREEN}Pretrained Model (Green boxes - After Korean Wikipedia):{Colors.ENDC}")
+    print(f"  • Korean: Should show IMPROVED fluency and vocabulary")
+    print(f"  • Better Korean sentence structure")
+    print(f"  • Weak instruction-following (only learned language, not how to follow instructions)")
+    print(f"  • English: Should REMAIN strong (no catastrophic forgetting)")
+    
+    print(f"\n{Colors.CYAN}Finetuned Model (Cyan boxes - After Instruction Tuning):{Colors.ENDC}")
+    print(f"  • Korean: Should be FLUENT with GOOD instruction-following")
+    print(f"  • More structured and complete responses")
+    print(f"  • Directly answers questions")
+    print(f"  • English: Should REMAIN strong")
+    
+    print(f"\n{Colors.YELLOW}Key Progression to Observe:{Colors.ENDC}")
+    print(f"  📊 Korean Quality:    {Colors.RED}Poor{Colors.ENDC} → {Colors.GREEN}Better{Colors.ENDC} → {Colors.CYAN}Best{Colors.ENDC}")
+    print(f"  📊 Instruction:       {Colors.RED}Weak{Colors.ENDC} → {Colors.GREEN}Weak{Colors.ENDC} → {Colors.CYAN}Strong{Colors.ENDC}")
+    print(f"  📊 English Quality:   {Colors.RED}Good{Colors.ENDC} → {Colors.GREEN}Good{Colors.ENDC} → {Colors.CYAN}Good{Colors.ENDC}")
+    print(f"  📊 Repetition:        {Colors.RED}High{Colors.ENDC} → {Colors.GREEN}Medium{Colors.ENDC} → {Colors.CYAN}Low{Colors.ENDC}")
+    
+    print(f"\n{Colors.YELLOW}This demonstrates:{Colors.ENDC}")
+    print(f"  ✓ Continued pretraining successfully teaches new language (Korean)")
+    print(f"  ✓ Instruction tuning teaches how to follow instructions in the new language")
+    print(f"  ✓ English capability is preserved throughout (no catastrophic forgetting)")
+    print(f"  ✓ Both Wikipedia and Instruction tasks show English preservation")
+    print(f"  ✓ Two-stage approach is necessary: language first, then instruction-following")
+    print(f"\n{Colors.CYAN}💡 Note: Compare the English tests (Tests 2 & 5) across all three models.")
+    print(f"All three should perform similarly well, proving no English degradation.{Colors.ENDC}")
+    print()
+
+if __name__ == "__main__":
+    main()

chapter8/continued-pretraining/continued-pretrain.py

@@ -0,0 +1,645 @@
+# -*- coding: utf-8 -*-
+"""Continued pretraining - Korean + Unsloth.ipynb
+
+Automatically generated by Colab.
+
+Original file is located at
+    https://colab.research.google.com/drive/1tEd1FrOXWMnCU9UIvdYhs61tkxdMuKZu
+
+To run this, press "*Runtime*" and press "*Run all*" on a **free** Tesla T4 Google Colab instance!
+<div class="align-center">
+  <a href="https://github.com/unslothai/unsloth"><img src="https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png" width="115"></a>
+  <a href="https://discord.gg/u54VK8m8tk"><img src="https://github.com/unslothai/unsloth/raw/main/images/Discord button.png" width="145"></a>
+  <a href="https://ko-fi.com/unsloth"><img src="https://github.com/unslothai/unsloth/raw/main/images/Kofi button.png" width="145"></a></a> Join Discord if you need help + ⭐ <i>Star us on <a href="https://github.com/unslothai/unsloth">Github</a> </i> ⭐
+</div>
+
+To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://github.com/unslothai/unsloth#installation-instructions---conda).
+
+You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save) (eg for Llama.cpp).
+
+We will use the Korean subset of the [Wikipedia dataset](https://huggingface.co/datasets/wikimedia/wikipedia) to first continually pretrain Mistral v3, then use the [Alpaca GPT4 Dataset](https://huggingface.co/datasets/FreedomIntelligence/alpaca-gpt4-korean) translated into Korean to further finetune the model to let it follow instructions in Korean.
+"""
+
+# ============================================================================
+# IMPORTS
+# ============================================================================
+import os
+import argparse
+
+
+# ============================================================================
+# 命令行参数（默认值与原始脚本硬编码值完全一致，保证行为不变）
+# ============================================================================
+def parse_args():
+    parser = argparse.ArgumentParser(
+        description="韩语 Mistral 继续预训练 + 指令微调（Unsloth / LoRA）。"
+                    "先用韩语维基百科做继续预训练注入韩语能力，再用韩语 Alpaca 数据做 SFT。",
+        formatter_class=argparse.ArgumentDefaultsHelpFormatter,
+    )
+
+    # 基础模型
+    parser.add_argument("--base_model", type=str, default="unsloth/mistral-7b-v0.3",
+                        help="基础模型名称（HuggingFace / Unsloth 仓库名）")
+    parser.add_argument("--max_seq_len", type=int, default=2048,
+                        help="最大序列长度")
+    parser.add_argument("--no_4bit", action="store_true",
+                        help="关闭 4bit 量化加载（默认开启 4bit 以节省显存）")
+
+    # LoRA 配置
+    parser.add_argument("--lora_rank", type=int, default=128,
+                        help="LoRA 秩 r（继续预训练建议较大，如 128）")
+    parser.add_argument("--lora_alpha", type=int, default=32,
+                        help="LoRA alpha")
+
+    # 数据集
+    parser.add_argument("--wiki_dataset", type=str, default="wikimedia/wikipedia",
+                        help="继续预训练用的维基百科数据集")
+    parser.add_argument("--wiki_config", type=str, default="20231101.ko",
+                        help="维基百科数据集的语言/版本配置（默认韩语快照）")
+    parser.add_argument("--wiki_train_size", type=float, default=0.05,
+                        help="维基百科数据集抽样比例（0~1，默认取 5%% 以加速训练）")
+    parser.add_argument("--alpaca_dataset", type=str,
+                        default="FreedomIntelligence/alpaca-gpt4-korean",
+                        help="指令微调（SFT）用的韩语 Alpaca 数据集")
+
+    # 训练超参数
+    parser.add_argument("--pretrain_epochs", type=int, default=1,
+                        help="继续预训练阶段的训练轮数")
+    parser.add_argument("--pretrain_max_steps", type=int, default=-1,
+                        help="继续预训练最大步数（-1 表示不限制，按 epoch 训练）")
+    parser.add_argument("--sft_epochs", type=int, default=2,
+                        help="指令微调阶段的训练轮数")
+    parser.add_argument("--sft_max_steps", type=int, default=-1,
+                        help="指令微调最大步数（-1 表示不限制，按 epoch 训练）")
+
+    # 输出目录
+    parser.add_argument("--pretrain_output_dir", type=str, default="outputs_pretrain",
+                        help="继续预训练的检查点目录")
+    parser.add_argument("--sft_output_dir", type=str, default="outputs_sft",
+                        help="指令微调的检查点目录")
+    parser.add_argument("--pretrained_save_dir", type=str, default="lora_model_pretrained",
+                        help="继续预训练后保存的 LoRA 模型目录")
+    parser.add_argument("--final_save_dir", type=str, default="lora_model",
+                        help="指令微调后保存的最终 LoRA 模型目录")
+
+    return parser.parse_args()
+
+
+# 先解析参数（放在重型导入之前，这样 `--help` 无需 GPU / Unsloth 也能运行）
+args = parse_args()
+
+import torch
+from unsloth import FastLanguageModel, is_bfloat16_supported, UnslothTrainer, UnslothTrainingArguments
+from transformers import TrainingArguments, TextStreamer
+from datasets import load_dataset
+
+# ANSI color codes for colored output
+class Colors:
+    HEADER = '\033[95m'
+    BLUE = '\033[94m'
+    CYAN = '\033[96m'
+    GREEN = '\033[92m'
+    YELLOW = '\033[93m'
+    RED = '\033[91m'
+    ENDC = '\033[0m'
+    BOLD = '\033[1m'
+    UNDERLINE = '\033[4m'
+
+def print_section(title, color=Colors.CYAN):
+    """Print a colored section header"""
+    print(f"\n{color}{Colors.BOLD}{'='*70}")
+    print(f"{title}")
+    print(f"{'='*70}{Colors.ENDC}\n")
+
+# ============================================================================
+# MODEL SETUP
+# ============================================================================
+print_section("🚀 LOADING MODEL", Colors.BLUE)
+
+max_seq_length = args.max_seq_len # Choose any! We auto support RoPE Scaling internally!
+dtype = None # None for auto detection. Float16 for Tesla T4, V100, Bfloat16 for Ampere+
+load_in_4bit = not args.no_4bit # Use 4bit quantization to reduce memory usage. Can be False.
+
+# 4bit pre quantized models we support for 4x faster downloading + no OOMs.
+fourbit_models = [
+    "unsloth/mistral-7b-v0.3-bnb-4bit",      # New Mistral v3 2x faster!
+    "unsloth/mistral-7b-instruct-v0.3-bnb-4bit",
+    "unsloth/llama-3-8b-bnb-4bit",           # Llama-3 15 trillion tokens model 2x faster!
+    "unsloth/llama-3-8b-Instruct-bnb-4bit",
+    "unsloth/llama-3-70b-bnb-4bit",
+    "unsloth/Phi-3-mini-4k-instruct",        # Phi-3 2x faster!
+    "unsloth/Phi-3-medium-4k-instruct",
+    "unsloth/mistral-7b-bnb-4bit",
+    "unsloth/gemma-7b-bnb-4bit",             # Gemma 2.2x faster!
+] # More models at https://huggingface.co/unsloth
+
+print(f"{Colors.GREEN}Loading {args.base_model}...{Colors.ENDC}")
+model, tokenizer = FastLanguageModel.from_pretrained(
+    model_name = args.base_model, # Choose ANY! eg teknium/OpenHermes-2.5-Mistral-7B
+    max_seq_length = max_seq_length,
+    dtype = dtype,
+    load_in_4bit = load_in_4bit,
+    # token = "hf_...", # use one if using gated models like meta-llama/Llama-2-7b-hf
+)
+
+print_section("⚙️ ADDING LORA ADAPTERS", Colors.BLUE)
+print(f"{Colors.GREEN}Adding LoRA adapters - only updating 1-10% of parameters!")
+print(f"Including embed_tokens and lm_head for continual pretraining{Colors.ENDC}")
+
+model = FastLanguageModel.get_peft_model(
+    model,
+    r = args.lora_rank, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128
+    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
+                      "gate_proj", "up_proj", "down_proj",
+                      "embed_tokens", "lm_head",], # Add for continual pretraining
+    lora_alpha = args.lora_alpha,
+    lora_dropout = 0, # Supports any, but = 0 is optimized
+    bias = "none",    # Supports any, but = "none" is optimized
+    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
+    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
+    random_state = 3407,
+    use_rslora = True,   # We support rank stabilized LoRA
+    loftq_config = None, # And LoftQ
+)
+
+# ============================================================================
+# TESTING ORIGINAL MODEL (BASELINE)
+# ============================================================================
+print_section("🧪 TESTING ORIGINAL MODEL (BASELINE)", Colors.CYAN)
+print(f"{Colors.YELLOW}Testing the base Mistral model BEFORE any training")
+print(f"This establishes baseline for Korean and English capabilities{Colors.ENDC}")
+
+FastLanguageModel.for_inference(model)
+text_streamer = TextStreamer(tokenizer)
+
+# Prepare prompts (define them early)
+_wikipedia_prompt = """Wikipedia Article
+### Title: {}
+
+### Article:
+{}"""
+
+wikipedia_prompt_korean = """위키피디아 기사
+### 제목: {}
+
+### 기사:
+{}"""
+
+_alpaca_prompt_english = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+### Instruction:
+{}
+
+### Response:
+{}"""
+
+alpaca_prompt_korean = """다음은 작업을 설명하는 명령입니다. 요청을 적절하게 완료하는 응답을 작성하세요.
+
+### 지침:
+{}
+
+### 응답:
+{}"""
+
+# Test 1: Korean Wikipedia (baseline - should be poor)
+print(f"\n{Colors.BOLD}Test 1: Korean Wikipedia Article (인공지능){Colors.ENDC}")
+print("="*70)
+test_prompt = wikipedia_prompt_korean.format("인공지능", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 2: English Wikipedia (baseline - should be good)
+print(f"\n{Colors.BOLD}Test 2: English Wikipedia Article (Artificial Intelligence){Colors.ENDC}")
+print("="*70)
+test_prompt = _wikipedia_prompt.format("Artificial Intelligence", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 3: Korean Instruction (baseline - should be poor)
+print(f"\n{Colors.BOLD}Test 3: Korean Instruction (Korean Culture){Colors.ENDC}")
+print("="*70)
+test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 4: English Instruction (baseline - should be good)
+print(f"\n{Colors.BOLD}Test 4: English Instruction (American Culture){Colors.ENDC}")
+print("="*70)
+test_prompt = _alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 5: Korean Instruction - Seoul (baseline - should be poor)
+print(f"\n{Colors.BOLD}Test 5: Korean Instruction (Korean Geography){Colors.ENDC}")
+print("="*70)
+test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+print(f"{Colors.GREEN}✓ Baseline testing complete. Original model should be good at English but poor at Korean.{Colors.ENDC}\n")
+
+# ============================================================================
+# DATA PREPARATION
+# ============================================================================
+print_section("📚 DATA PREPARATION - WIKIPEDIA KOREAN DATASET", Colors.CYAN)
+print(f"{Colors.YELLOW}Loading Korean Wikipedia dataset...")
+print(f"We'll use 5% of the dataset to speed up training{Colors.ENDC}")
+
+# Use the prompts already defined above
+EOS_TOKEN = tokenizer.eos_token # Must add EOS_TOKEN
+def formatting_prompts_func_wiki(examples):
+    titles = examples["title"]
+    texts  = examples["text"]
+    outputs = []
+    for title, text in zip(titles, texts):
+        # Must add EOS_TOKEN, otherwise your generation will go on forever!
+        text = wikipedia_prompt_korean.format(title, text) + EOS_TOKEN
+        outputs.append(text)
+    return { "text" : outputs, }
+
+dataset = load_dataset(args.wiki_dataset, args.wiki_config, split = "train",)
+# We select 5% of the data to make training faster!
+dataset = dataset.train_test_split(train_size = args.wiki_train_size)["train"]
+dataset = dataset.map(formatting_prompts_func_wiki, batched = True,)
+
+print(f"{Colors.GREEN}✓ Wikipedia dataset loaded: {len(dataset)} examples{Colors.ENDC}")
+
+print_section("📚 DATA PREPARATION - ALPACA KOREAN DATASET", Colors.CYAN)
+print(f"{Colors.YELLOW}Loading Alpaca GPT4 Korean dataset for instruction finetuning...{Colors.ENDC}")
+
+alpaca_dataset = load_dataset(args.alpaca_dataset, split = "train")
+
+# Use the prompts already defined above
+def formatting_prompts_func_alpaca(conversations):
+    texts = []
+    conversations = conversations["conversations"]
+    for convo in conversations:
+        # Must add EOS_TOKEN, otherwise your generation will go on forever!
+        text = alpaca_prompt_korean.format(convo[0]["value"], convo[1]["value"]) + EOS_TOKEN
+        texts.append(text)
+    return { "text" : texts, }
+
+alpaca_dataset = alpaca_dataset.map(formatting_prompts_func_alpaca, batched = True,)
+
+print(f"{Colors.GREEN}✓ Alpaca dataset loaded: {len(alpaca_dataset)} examples{Colors.ENDC}")
+print(f"\nExample from Alpaca dataset:")
+print(alpaca_dataset[0])
+
+# ============================================================================
+# CONTINUED PRETRAINING
+# ============================================================================
+print_section("🎯 CONTINUED PRETRAINING ON KOREAN WIKIPEDIA", Colors.GREEN)
+print(f"{Colors.YELLOW}Training the model on Korean Wikipedia to learn the language...")
+print(f"Using embedding_learning_rate (1e-5) smaller than learning_rate (5e-5)")
+print(f"💾 Checkpoints will be saved every 100 steps to: outputs_pretrain/")
+print(f"Only the 5 most recent checkpoints will be kept{Colors.ENDC}")
+
+# Set WandB project for pretraining
+os.environ["WANDB_PROJECT"] = "unsloth-continued-pretraining"
+
+trainer = UnslothTrainer(
+    model = model,
+    tokenizer = tokenizer,
+    train_dataset = dataset,
+    dataset_text_field = "text",
+    max_seq_length = max_seq_length,
+    dataset_num_proc = 2,
+
+    args = UnslothTrainingArguments(
+        per_device_train_batch_size = 2,
+        gradient_accumulation_steps = 8,
+
+        # Use warmup_ratio and num_train_epochs for longer runs!
+        max_steps = args.pretrain_max_steps,
+        warmup_steps = 10,
+        warmup_ratio = 0.1,
+        num_train_epochs = args.pretrain_epochs,
+
+        # Select a 2 to 10x smaller learning rate for the embedding matrices!
+        learning_rate = 5e-5,
+        embedding_learning_rate = 1e-5,
+
+        fp16 = not is_bfloat16_supported(),
+        bf16 = is_bfloat16_supported(),
+        logging_steps = 1,
+        optim = "adamw_8bit",
+        weight_decay = 0.01,
+        lr_scheduler_type = "linear",
+        seed = 42,
+        output_dir = args.pretrain_output_dir,
+        
+        # Checkpoint saving
+        save_strategy = "steps",
+        save_steps = 100,
+        # Keep only the 5 most recent checkpoints, as the banner above
+        # promises. Without this HF keeps every one, and a ~2000-step run
+        # writes ~20 LoRA+optimizer checkpoints (fills a free-Colab disk).
+        save_total_limit = 5,
+        
+        # WandB configuration
+        report_to = "wandb",
+        run_name = "korean-mistral-pretrain",
+    ),
+)
+
+print_section("💾 MEMORY STATS - BEFORE PRETRAINING", Colors.YELLOW)
+gpu_stats = torch.cuda.get_device_properties(0)
+start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
+max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
+print(f"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.")
+print(f"{start_gpu_memory} GB of memory reserved.")
+
+trainer_stats = trainer.train()
+
+# Finish wandb run for pretraining
+import wandb
+if wandb.run is not None:
+    wandb.finish()
+    print(f"{Colors.CYAN}✓ Finished wandb run for pretraining{Colors.ENDC}")
+
+print_section("💾 SAVING PRETRAINED MODEL", Colors.GREEN)
+model.save_pretrained(args.pretrained_save_dir) # Local saving
+tokenizer.save_pretrained(args.pretrained_save_dir)
+print(f"{Colors.GREEN}✓ Model saved to: {args.pretrained_save_dir}/{Colors.ENDC}")
+
+# ============================================================================
+# TESTING PRETRAINED MODEL
+# ============================================================================
+print_section("🧪 TESTING PRETRAINED MODEL (AFTER KOREAN PRETRAINING)", Colors.CYAN)
+print(f"{Colors.YELLOW}Testing after Korean pretraining - before instruction finetuning")
+print(f"Korean should improve, English should remain strong{Colors.ENDC}")
+
+# Test the pretrained model before instruction finetuning
+FastLanguageModel.for_inference(model) # Enable native 2x faster inference
+text_streamer = TextStreamer(tokenizer)
+
+# Test 1: Korean Wikipedia (same as baseline)
+print(f"\n{Colors.BOLD}Test 1: Korean Wikipedia Article (인공지능){Colors.ENDC}")
+print("="*70)
+test_prompt = wikipedia_prompt_korean.format("인공지능", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 2: English Wikipedia (same as baseline)
+print(f"\n{Colors.BOLD}Test 2: English Wikipedia Article (Artificial Intelligence){Colors.ENDC}")
+print("="*70)
+test_prompt = _wikipedia_prompt.format("Artificial Intelligence", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 3: Korean Instruction (same as baseline - should improve but not follow perfectly yet)
+print(f"\n{Colors.BOLD}Test 3: Korean Instruction (Korean Culture){Colors.ENDC}")
+print("="*70)
+test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 4: English Instruction (same as baseline)
+print(f"\n{Colors.BOLD}Test 4: English Instruction (American Culture){Colors.ENDC}")
+print("="*70)
+test_prompt = _alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 5: Korean Instruction - Seoul (should improve but not perfect yet)
+print(f"\n{Colors.BOLD}Test 5: Korean Instruction (Korean Geography){Colors.ENDC}")
+print("="*70)
+test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+print(f"{Colors.GREEN}✓ Pretrained model testing complete.")
+print(f"{Colors.CYAN}💡 Korean should be significantly improved, English should remain strong.")
+print(f"Instruction following may improve but not perfect yet - that's what SFT is for.{Colors.ENDC}\n")
+
+# ============================================================================
+# INSTRUCTION FINETUNING
+# ============================================================================
+print_section("🎓 INSTRUCTION FINETUNING ON ALPACA KOREAN", Colors.GREEN)
+print(f"{Colors.YELLOW}Now finetuning the model to follow Korean instructions...")
+print(f"Using the Alpaca GPT4 dataset translated to Korean")
+print(f"💾 Checkpoints will be saved every 100 steps to: outputs_sft/")
+print(f"Only the 5 most recent checkpoints will be kept{Colors.ENDC}")
+
+# Set WandB project for finetuning
+os.environ["WANDB_PROJECT"] = "unsloth-continued-finetuning"
+
+trainer = UnslothTrainer(
+    model = model,
+    tokenizer = tokenizer,
+    train_dataset = alpaca_dataset,
+    dataset_text_field = "text",
+    max_seq_length = max_seq_length,
+    dataset_num_proc = 8,
+
+    args = UnslothTrainingArguments(
+        per_device_train_batch_size = 2,
+        gradient_accumulation_steps = 8,
+
+        # Use num_train_epochs and warmup_ratio for longer runs!
+        max_steps = args.sft_max_steps,
+        warmup_steps = 10,
+        warmup_ratio = 0.1,
+        num_train_epochs = args.sft_epochs,
+
+        # Select a 2 to 10x smaller learning rate for the embedding matrices!
+        learning_rate = 5e-5,
+        embedding_learning_rate = 1e-5,
+
+        fp16 = not is_bfloat16_supported(),
+        bf16 = is_bfloat16_supported(),
+        logging_steps = 1,
+        optim = "adamw_8bit",
+        weight_decay = 0.00,
+        lr_scheduler_type = "linear",
+        seed = 42,
+        output_dir = args.sft_output_dir,
+        
+        # Checkpoint saving
+        save_strategy = "steps",
+        save_steps = 100,
+        # Keep only the 5 most recent checkpoints, as the banner above
+        # promises. Without this HF keeps every one, and a ~2000-step run
+        # writes ~20 LoRA+optimizer checkpoints (fills a free-Colab disk).
+        save_total_limit = 5,
+        
+        # WandB configuration
+        report_to = "wandb",
+        run_name = "korean-mistral-finetune",
+    ),
+)
+
+trainer_stats = trainer.train()
+
+# Finish wandb run for SFT
+if wandb.run is not None:
+    wandb.finish()
+    print(f"{Colors.CYAN}✓ Finished wandb run for SFT{Colors.ENDC}")
+
+print_section("📊 FINAL MEMORY AND TIME STATS", Colors.YELLOW)
+used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
+used_memory_for_lora = round(used_memory - start_gpu_memory, 3)
+used_percentage = round(used_memory         /max_memory*100, 3)
+lora_percentage = round(used_memory_for_lora/max_memory*100, 3)
+print(f"{trainer_stats.metrics['train_runtime']} seconds used for training.")
+print(f"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.")
+print(f"Peak reserved memory = {used_memory} GB.")
+print(f"Peak reserved memory for training = {used_memory_for_lora} GB.")
+print(f"Peak reserved memory % of max memory = {used_percentage} %.")
+print(f"Peak reserved memory for training % of max memory = {lora_percentage} %.")
+
+# ============================================================================
+# INFERENCE - TESTING FINETUNED MODEL
+# ============================================================================
+print_section("🎯 INFERENCE - TESTING FINETUNED MODEL", Colors.CYAN)
+print(f"{Colors.YELLOW}Testing the instruction-finetuned model - should follow instructions in Korean AND English{Colors.ENDC}")
+
+FastLanguageModel.for_inference(model) # Enable native 2x faster inference
+text_streamer = TextStreamer(tokenizer)
+
+# Test 1: Korean Wikipedia (same as baseline)
+print(f"\n{Colors.BOLD}Test 1: Korean Wikipedia Article (인공지능){Colors.ENDC}")
+print("="*70)
+test_prompt = wikipedia_prompt_korean.format("인공지능", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 2: English Wikipedia (same as baseline)
+print(f"\n{Colors.BOLD}Test 2: English Wikipedia Article (Artificial Intelligence){Colors.ENDC}")
+print("="*70)
+test_prompt = _wikipedia_prompt.format("Artificial Intelligence", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 3: Korean Instruction (same as baseline - should follow WELL now)
+print(f"\n{Colors.BOLD}Test 3: Korean Instruction (Korean Culture){Colors.ENDC}")
+print("="*70)
+test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 4: English Instruction (same as baseline)
+print(f"\n{Colors.BOLD}Test 4: English Instruction (American Culture){Colors.ENDC}")
+print("="*70)
+test_prompt = _alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+# Test 5: Korean Instruction - Seoul (should follow WELL now)
+print(f"\n{Colors.BOLD}Test 5: Korean Instruction (Korean Geography){Colors.ENDC}")
+print("="*70)
+test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
+inputs = tokenizer([test_prompt], return_tensors = "pt").to("cuda")
+_ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 150, use_cache = True)
+print("="*70 + "\n")
+
+print(f"{Colors.GREEN}✓ Finetuned model testing complete!")
+print(f"{Colors.CYAN}💡 Model should now follow instructions well in both Korean AND English.{Colors.ENDC}\n")
+
+# ============================================================================
+# SAVING FINAL MODEL
+# ============================================================================
+print_section("💾 SAVING FINAL FINETUNED MODEL", Colors.GREEN)
+
+model.save_pretrained(args.final_save_dir) # Local saving
+tokenizer.save_pretrained(args.final_save_dir)
+print(f"{Colors.GREEN}✓ Final model saved to: {args.final_save_dir}/{Colors.ENDC}")
+# model.push_to_hub("your_name/lora_model", token = "...") # Online saving
+# tokenizer.push_to_hub("your_name/lora_model", token = "...") # Online saving
+
+print(f"\n{Colors.CYAN}Note: This only saves LoRA adapters, not the full model.")
+print(f"For 16bit or GGUF formats, see the export options below.{Colors.ENDC}")
+
+# ============================================================================
+# LOADING SAVED MODEL (OPTIONAL)
+# ============================================================================
+"""
+print_section("📥 LOADING SAVED MODEL", Colors.BLUE)
+
+# Set to True to test loading
+if False:
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name = "lora_model", # YOUR MODEL YOU USED FOR TRAINING
+        max_seq_length = max_seq_length,
+        dtype = dtype,
+        load_in_4bit = load_in_4bit,
+    )
+    FastLanguageModel.for_inference(model) # Enable native 2x faster inference
+
+    inputs = tokenizer(
+    [
+        alpaca_prompt.format(
+            # "Describe the planet Earth extensively.", # instruction
+            "지구를 광범위하게 설명하세요.",
+            "", # output - leave this blank for generation!
+        ),
+    ], return_tensors = "pt").to("cuda")
+
+    text_streamer = TextStreamer(tokenizer)
+    _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 256)
+"""
+
+# ============================================================================
+# EXPORT OPTIONS
+# ============================================================================
+print_section("📦 EXPORT OPTIONS", Colors.BLUE)
+print(f"{Colors.YELLOW}Various export formats available (currently disabled):{Colors.ENDC}")
+print("• Float16 merged model")
+print("• 4bit merged model")
+print("• LoRA adapters only")
+print("• GGUF format for llama.cpp")
+
+# Merge to 16bit
+if False: 
+    print_section("💾 EXPORTING TO FLOAT16", Colors.GREEN)
+    model.save_pretrained_merged("model", tokenizer, save_method = "merged_16bit",)
+    # model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_16bit", token = "")
+
+# Merge to 4bit
+if False: 
+    print_section("💾 EXPORTING TO 4BIT", Colors.GREEN)
+    model.save_pretrained_merged("model", tokenizer, save_method = "merged_4bit",)
+    # model.push_to_hub_merged("hf/model", tokenizer, save_method = "merged_4bit", token = "")
+
+# Just LoRA adapters
+if False: 
+    print_section("💾 EXPORTING LORA ADAPTERS", Colors.GREEN)
+    model.save_pretrained_merged("model", tokenizer, save_method = "lora",)
+    # model.push_to_hub_merged("hf/model", tokenizer, save_method = "lora", token = "")
+
+# GGUF exports
+if False:
+    print_section("💾 EXPORTING TO GGUF Q8_0", Colors.GREEN)
+    model.save_pretrained_gguf("model", tokenizer,)
+    # model.push_to_hub_gguf("hf/model", tokenizer, token = "")
+
+if False:
+    print_section("💾 EXPORTING TO GGUF F16", Colors.GREEN)
+    model.save_pretrained_gguf("model", tokenizer, quantization_method = "f16")
+    # model.push_to_hub_gguf("hf/model", tokenizer, quantization_method = "f16", token = "")
+
+if False:
+    print_section("💾 EXPORTING TO GGUF Q4_K_M", Colors.GREEN)
+    model.save_pretrained_gguf("model", tokenizer, quantization_method = "q4_k_m")
+    # model.push_to_hub_gguf("hf/model", tokenizer, quantization_method = "q4_k_m", token = "")
+
+print_section("✅ TRAINING COMPLETE!", Colors.GREEN)
+print(f"{Colors.BOLD}Your Korean Mistral model is ready to use!{Colors.ENDC}")
+print(f"\n{Colors.CYAN}For more information:{Colors.ENDC}")
+print("• Discord: https://discord.gg/u54VK8m8tk")
+print("• GitHub: https://github.com/unslothai/unsloth")
+print("• Documentation: https://docs.unsloth.ai")

chapter8/continued-pretraining/evaluate_model.py

@@ -0,0 +1,285 @@
+# -*- coding: utf-8 -*-
+"""
+Evaluation script for Korean Mistral continued-pretrained models
+Loads saved LoRA adapters and evaluates on Korean and English tasks
+"""
+
+import os
+import argparse
+
+# 说明：unsloth / torch / transformers 等重型依赖在函数内按需导入，
+# 这样 `python evaluate_model.py --help` 无需 GPU 环境即可查看参数。
+
+# ANSI color codes for colored output
+class Colors:
+    HEADER = '\033[95m'
+    BLUE = '\033[94m'
+    CYAN = '\033[96m'
+    GREEN = '\033[92m'
+    YELLOW = '\033[93m'
+    RED = '\033[91m'
+    ENDC = '\033[0m'
+    BOLD = '\033[1m'
+    UNDERLINE = '\033[4m'
+
+def print_section(title, color=Colors.CYAN):
+    """Print a colored section header"""
+    print(f"\n{color}{Colors.BOLD}{'='*70}")
+    print(f"{title}")
+    print(f"{'='*70}{Colors.ENDC}\n")
+
+def load_model(model_path, max_seq_length=2048, dtype=None, load_in_4bit=True):
+    """Load the saved LoRA model"""
+    from unsloth import FastLanguageModel
+    print_section(f"📥 LOADING MODEL FROM: {model_path}", Colors.BLUE)
+
+    print(f"{Colors.YELLOW}Loading model and tokenizer...{Colors.ENDC}")
+    model, tokenizer = FastLanguageModel.from_pretrained(
+        model_name = model_path,
+        max_seq_length = max_seq_length,
+        dtype = dtype,
+        load_in_4bit = load_in_4bit,
+    )
+    
+    FastLanguageModel.for_inference(model)  # Enable native 2x faster inference
+    print(f"{Colors.GREEN}✓ Model loaded successfully!{Colors.ENDC}")
+    
+    return model, tokenizer
+
+def run_evaluation(model, tokenizer, max_new_tokens=150, 
+                   temperature=0.7, top_p=0.9, use_sampling=False):
+    """Run all evaluation tests"""
+    from transformers import TextStreamer
+
+    print_section("🧪 RUNNING EVALUATION TESTS", Colors.CYAN)
+    print(f"{Colors.YELLOW}Generation Parameters:{Colors.ENDC}")
+    print(f"  • max_new_tokens: {max_new_tokens}")
+    if use_sampling:
+        print(f"  • temperature: {temperature}")
+        print(f"  • top_p: {top_p}")
+        print(f"  • Sampling: Enabled")
+    else:
+        print(f"  • Sampling: Disabled (greedy decoding)")
+    
+    text_streamer = TextStreamer(tokenizer, skip_special_tokens=True)
+    
+    # Define prompts
+    wikipedia_prompt_korean = """위키피디아 기사
+### 제목: {}
+
+### 기사:
+{}"""
+
+    wikipedia_prompt_english = """Wikipedia Article
+### Title: {}
+
+### Article:
+{}"""
+
+    alpaca_prompt_korean = """다음은 작업을 설명하는 명령입니다. 요청을 적절하게 완료하는 응답을 작성하세요.
+
+### 지침:
+{}
+
+### 응답:
+{}"""
+
+    alpaca_prompt_english = """Below is an instruction that describes a task. Write a response that appropriately completes the request.
+
+### Instruction:
+{}
+
+### Response:
+{}"""
+
+    # Prepare generation kwargs
+    gen_kwargs = {
+        "max_new_tokens": max_new_tokens,
+        "use_cache": True,
+    }
+    
+    if use_sampling:
+        gen_kwargs.update({
+            "do_sample": True,
+            "temperature": temperature,
+            "top_p": top_p,
+        })
+    
+    # Test 1: Korean Wikipedia Article
+    print(f"\n{Colors.BOLD}{'='*70}")
+    print(f"Test 1: Korean Wikipedia Article - Artificial Intelligence (인공지능)")
+    print(f"{'='*70}{Colors.ENDC}")
+    print(f"{Colors.CYAN}Prompt (Translation): Wikipedia Article / Title: Artificial Intelligence / Article:{Colors.ENDC}\n")
+    
+    test_prompt = wikipedia_prompt_korean.format("인공지능", "")
+    inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
+    print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
+    _ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
+    print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
+    
+    # Test 2: English Wikipedia Article
+    print(f"\n{Colors.BOLD}{'='*70}")
+    print(f"Test 2: English Wikipedia Article - Artificial Intelligence")
+    print(f"{'='*70}{Colors.ENDC}\n")
+    
+    test_prompt = wikipedia_prompt_english.format("Artificial Intelligence", "")
+    inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
+    print(f"{Colors.GREEN}[ENGLISH OUTPUT]{Colors.ENDC}")
+    _ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
+    print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
+    
+    # Test 3: Korean Instruction (Kimchi)
+    print(f"\n{Colors.BOLD}{'='*70}")
+    print(f"Test 3: Korean Instruction - Explain about Kimchi")
+    print(f"{'='*70}{Colors.ENDC}")
+    print(f"{Colors.CYAN}Prompt (Translation): Instruction: Explain about kimchi, a traditional Korean food. / Response:{Colors.ENDC}\n")
+    
+    test_prompt = alpaca_prompt_korean.format("한국의 전통 음식인 김치에 대해 설명하세요.", "")
+    inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
+    print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
+    _ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
+    print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
+    
+    # Test 4: English Instruction (Thanksgiving)
+    print(f"\n{Colors.BOLD}{'='*70}")
+    print(f"Test 4: English Instruction - Explain about Thanksgiving Turkey")
+    print(f"{'='*70}{Colors.ENDC}\n")
+    
+    test_prompt = alpaca_prompt_english.format("Explain about Thanksgiving turkey, a traditional American food.", "")
+    inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
+    print(f"{Colors.GREEN}[ENGLISH OUTPUT]{Colors.ENDC}")
+    _ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
+    print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
+    
+    # Additional Korean tests
+    print(f"\n{Colors.BOLD}{'='*70}")
+    print(f"Test 5: Korean Instruction - Explain about Seoul")
+    print(f"{'='*70}{Colors.ENDC}")
+    print(f"{Colors.CYAN}Prompt (Translation): Instruction: Briefly introduce Seoul, the capital of South Korea. / Response:{Colors.ENDC}\n")
+    
+    test_prompt = alpaca_prompt_korean.format("대한민국의 수도인 서울에 대해 간단히 소개해주세요.", "")
+    inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
+    print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
+    _ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
+    print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
+    
+    # Test 6: Korean Instruction (K-pop)
+    print(f"\n{Colors.BOLD}{'='*70}")
+    print(f"Test 6: Korean Instruction - Explain about K-pop")
+    print(f"{'='*70}{Colors.ENDC}")
+    print(f"{Colors.CYAN}Prompt (Translation): Instruction: Explain what K-pop is. / Response:{Colors.ENDC}\n")
+    
+    test_prompt = alpaca_prompt_korean.format("K-pop이 무엇인지 설명해주세요.", "")
+    inputs = tokenizer([test_prompt], return_tensors="pt").to("cuda")
+    print(f"{Colors.GREEN}[KOREAN OUTPUT]{Colors.ENDC}")
+    _ = model.generate(**inputs, streamer=text_streamer, **gen_kwargs)
+    print(f"\n{Colors.BOLD}{'='*70}{Colors.ENDC}\n")
+    
+    print_section("✅ EVALUATION COMPLETE", Colors.GREEN)
+
+def main():
+    parser = argparse.ArgumentParser(description="Evaluate Korean Mistral LoRA models")
+    parser.add_argument(
+        "--model_path",
+        type=str,
+        default="lora_model",
+        help="Path to the saved LoRA model (default: lora_model)"
+    )
+    parser.add_argument(
+        "--pretrained",
+        action="store_true",
+        help="Load the pretrained model (before SFT) instead of final model"
+    )
+    parser.add_argument(
+        "--max_seq_length",
+        type=int,
+        default=2048,
+        help="Maximum sequence length (default: 2048)"
+    )
+    parser.add_argument(
+        "--load_in_4bit",
+        action="store_true",
+        default=True,
+        help="Load model in 4-bit quantization (default: True)"
+    )
+    parser.add_argument(
+        "--max_new_tokens",
+        type=int,
+        default=150,
+        help="Maximum number of tokens to generate (default: 150)"
+    )
+    parser.add_argument(
+        "--use_sampling",
+        action="store_true",
+        help="Use sampling instead of greedy decoding"
+    )
+    parser.add_argument(
+        "--temperature",
+        type=float,
+        default=0.7,
+        help="Sampling temperature (default: 0.7, only used with --use_sampling)"
+    )
+    parser.add_argument(
+        "--top_p",
+        type=float,
+        default=0.9,
+        help="Top-p sampling parameter (default: 0.9, only used with --use_sampling)"
+    )
+    
+    args = parser.parse_args()
+    
+    # Determine model path
+    if args.pretrained:
+        model_path = "lora_model_pretrained"
+        print(f"{Colors.YELLOW}Loading PRETRAINED model (before instruction finetuning){Colors.ENDC}")
+    else:
+        model_path = args.model_path
+        print(f"{Colors.YELLOW}Loading FINETUNED model (after instruction finetuning){Colors.ENDC}")
+    
+    # Check if model exists
+    if not os.path.exists(model_path):
+        print(f"{Colors.RED}Error: Model path '{model_path}' does not exist!{Colors.ENDC}")
+        print(f"{Colors.YELLOW}Make sure you've run the training script first.{Colors.ENDC}")
+        return
+    
+    print_section("🚀 KOREAN MISTRAL MODEL EVALUATION", Colors.HEADER)
+    
+    # Load model
+    model, tokenizer = load_model(
+        model_path=model_path,
+        max_seq_length=args.max_seq_length,
+        load_in_4bit=args.load_in_4bit
+    )
+    
+    # Display GPU info
+    import torch
+    print_section("💾 GPU MEMORY STATS", Colors.YELLOW)
+    gpu_stats = torch.cuda.get_device_properties(0)
+    reserved_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)
+    max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)
+    print(f"GPU: {gpu_stats.name}")
+    print(f"Max memory: {max_memory} GB")
+    print(f"Reserved memory: {reserved_memory} GB")
+    
+    # Run evaluation
+    run_evaluation(
+        model=model,
+        tokenizer=tokenizer,
+        max_new_tokens=args.max_new_tokens,
+        temperature=args.temperature,
+        top_p=args.top_p,
+        use_sampling=args.use_sampling
+    )
+    
+    print(f"\n{Colors.CYAN}{'='*70}")
+    print(f"💡 Tips:")
+    print(f"{'='*70}{Colors.ENDC}")
+    print(f"• Compare pretrained vs finetuned: Run with --pretrained flag")
+    print(f"• Adjust generation: Use --max_new_tokens")
+    print(f"• Enable sampling: Use --use_sampling --temperature 0.7 --top_p 0.9")
+    print(f"• Example: python evaluate_model.py --pretrained --max_new_tokens 300")
+    print()
+
+if __name__ == "__main__":
+    main()
+

chapter8/continued-pretraining/model_eval_results.md

@@ -0,0 +1,487 @@
+# Comprehensive Evaluation with Translations
+
+## Raw Evaluation Results
+
+```
+$ python compare_models.py
+🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning.
+Skipping import of cpp extensions due to incompatible torch version 2.8.0+cu128 for torchao version 0.14.0         Please see GitHub issue #2919 for more info
+🦥 Unsloth Zoo will now patch everything to make training faster!
+/venv/main/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'repr' attribute with value False was provided to the `Field()` function, which has no effect in the context it was used. 'repr' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
+  warnings.warn(
+/venv/main/lib/python3.10/site-packages/pydantic/_internal/_generate_schema.py:2249: UnsupportedFieldAttributeWarning: The 'frozen' attribute with value True was provided to the `Field()` function, which has no effect in the context it was used. 'frozen' is field-specific metadata, and can only be attached to a model field using `Annotated` metadata or by assignment. This may have happened because an `Annotated` type alias using the `type` statement was used, or if the `Field()` function was attached to a single member of a union type.
+  warnings.warn(
+
+================================================================================
+🔬 KOREAN MISTRAL 3-WAY MODEL COMPARISON
+================================================================================
+
+This script compares three model stages:
+  1. Baseline - Original Mistral (no Korean training)
+  2. Pretrained - After Korean Wikipedia training
+  3. Finetuned - After instruction tuning
+
+Generation settings: temperature=0.3, do_sample=True (no repetition_penalty)
+
+
+================================================================================
+📥 LOADING MODELS
+================================================================================
+
+Loading baseline model (original Mistral v0.3)...
+==((====))==  Unsloth 2025.10.4: Fast Mistral patching. Transformers: 4.56.2.
+   \\   /|    NVIDIA GeForce RTX 4090. Num GPUs = 1. Max memory: 23.647 GB. Platform: Linux.
+O^O/ \_/ \    Torch: 2.8.0+cu128. CUDA: 8.9. CUDA Toolkit: 12.8. Triton: 3.4.0
+\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False]
+ "-____-"     Free license: http://github.com/unslothai/unsloth
+Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
+✓ Baseline model loaded
+
+Loading pretrained model (after Korean pretraining)...
+==((====))==  Unsloth 2025.10.4: Fast Mistral patching. Transformers: 4.56.2.
+   \\   /|    NVIDIA GeForce RTX 4090. Num GPUs = 1. Max memory: 23.647 GB. Platform: Linux.
+O^O/ \_/ \    Torch: 2.8.0+cu128. CUDA: 8.9. CUDA Toolkit: 12.8. Triton: 3.4.0
+\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False]
+ "-____-"     Free license: http://github.com/unslothai/unsloth
+Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
+Unsloth: Will load lora_model_pretrained as a legacy tokenizer.
+Unsloth 2025.10.4 patched 32 layers with 32 QKV layers, 32 O layers and 32 MLP layers.
+✓ Pretrained model loaded
+
+Loading finetuned model (after instruction tuning)...
+==((====))==  Unsloth 2025.10.4: Fast Mistral patching. Transformers: 4.56.2.
+   \\   /|    NVIDIA GeForce RTX 4090. Num GPUs = 1. Max memory: 23.647 GB. Platform: Linux.
+O^O/ \_/ \    Torch: 2.8.0+cu128. CUDA: 8.9. CUDA Toolkit: 12.8. Triton: 3.4.0
+\        /    Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False]
+ "-____-"     Free license: http://github.com/unslothai/unsloth
+Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
+Unsloth: Will load lora_model as a legacy tokenizer.
+✓ Finetuned model loaded
+
+================================================================================
+🧪 RUNNING 3-WAY COMPARISONS
+================================================================================
+
+
+================================================================================
+Test 1: Korean Wikipedia - Artificial Intelligence (인공지능)
+================================================================================
+Prompt (Translation): Wikipedia Article / Title: Artificial Intelligence / Article:
+
+Generating from BASELINE model (original Mistral)...
+Generating from PRETRAINED model (after Korean training)...
+Generating from FINETUNED model (after instruction tuning)...
+
+┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
+│
+│ 인공지능(artificial intelligence, AI)은 인간의 지능을 모방하는 컴퓨터 프로그램이다. 인공지능은 인간의 지능을 모방하는 것이 아니라 인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인간의 지능을 모방하는 것이 아니
+   인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
+│
+│ 인공지능(人工智能, )은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인공지능은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이며, 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인간의 지적 능력을 모방하
+   컴퓨터 소프
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
+│
+│ 인공지능(AI)은 인간 지능이 필요한 작업을 수행할 수 있는 컴퓨터 시스템을 개발하는 것을 의미합니다. 인간의 지능을 모방하는 기계 또는 컴퓨터 프로그램을 만드는 것을 의미합니다. 이러한 작업에는 자연어 이해, 이미지 인
+  , 의사 결정 및 문제 해결 등이
+│
+└────────────────────────────────────────────────────────────────────┘
+
+
+================================================================================
+Test 2: English Wikipedia - Artificial Intelligence (Preservation Check)
+================================================================================
+Generating from BASELINE model (original Mistral)...
+Generating from PRETRAINED model (after Korean training)...
+Generating from FINETUNED model (after instruction tuning)...
+
+┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
+│
+│ Artificial intelligence (AI), sometimes called machine intelligence, is intelligence demonstrated by machines, in contrast to the natural intelligence (NI) displayed by humans and other animals. In computer science AI research is defined as the study of "intelligent agents": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals. Colloquially, the term "artificial intelligence" is applied when a machine mimics "cognitive" functions that humans associate with other human minds, such as "learning" and "problem solving".
+│
+│ As machines become increasingly capable, tasks considered to require "intelligence" are often removed from the definition, a phenomenon known as the AI effect, leading to
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
+│
+│ Artificial intelligence (AI) is intelligence exhibited by machines. In computer science, an AI is a machine that mimics human intelligence. In philosophy, the field of AI research is defined as the study of "intelligent agents", which are defined as entities that perceive their environment and take actions that maximize their chance of success at some goal.
+│
+│ AI research is highly interdisciplinary, since it spans the gap between many fields of computer science and other academic fields like philosophy, psychology, linguistics, physics, and biology. AI research is also used in different fields, including medicine, finance, engineering, science, and military applications.
+│
+│ The field was founded as an academic discipline in 195
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
+│
+│ Artificial intelligence (AI) is intelligence exhibited by computers. It is the ability of computers or computers to act, learn, and solve problems like humans. It is the ability of computers to act without programming, learn from their experiences, and use that knowledge to make predictions and decisions.
+│
+│ AI systems can be broken down into three types:
+│
+│ 1. Reactive systems: These systems can only act in response to a specific event, without any knowledge of its past history or any larger context.
+│
+│ 2. Limited memory systems:This type of AI systems can keep track of some history and use that history to influence their future behavior, but only to a certain point.
+│
+│ 3. Theories of mind systems:This
+│
+└────────────────────────────────────────────────────────────────────┘
+
+
+================================================================================
+Test 3: Korean Instruction - Explain Kimchi
+================================================================================
+Prompt (Translation): Instruction: Explain about kimchi, a traditional Korean food. / Response:
+
+Generating from BASELINE model (original Mistral)...
+Generating from PRETRAINED model (after Korean training)...
+Generating from FINETUNED model (after instruction tuning)...
+
+┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
+│
+│ 김치는 한국의 전통 음식이며, 육식 밥과 함께 먹는 것이 일반적입니다. 김치는 살기 높은 채소를 잡음료로 만들어 손쉽게 만들 수 있습니다. 김치는 다양한 식품에 사용되며, 살기 높은 채소를 잡음료로 만들어 손쉽게 만들
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
+│
+│ 김치는 한국의 전통 음식으로, 찹쌀가루와 칠면조를 섞어 만들어 삶아 낸 음식이다. 김치는 건강을 위해 좋은 음식으로 여겨지며, 섭취할 때마다 맛이 좋아진다.
+│
+│ 김치는 한국 전통 음식의 일부로 널리 알려져 있으며, 세계 각국에서 인기가
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
+│
+│ 김치는 한국의 전통 음식으로, 콩나물이나 양배추와 같은 채소를 삶은 후 쌀죽 또는 쌀로 만든 맛있는 간장 소스에 담근 것입니다. 이 소스는 쌀죽 또는 쌀로 만들어지며, 다양한 양념과 향신료로 만들어집니다. 이 소스는 채소
+   담아 담근
+│
+└────────────────────────────────────────────────────────────────────┘
+
+
+================================================================================
+Test 4: Korean Instruction - Introduce Seoul
+================================================================================
+Prompt (Translation): Instruction: Briefly introduce Seoul, the capital of South Korea. / Response:
+
+Generating from BASELINE model (original Mistral)...
+Generating from PRETRAINED model (after Korean training)...
+Generating from FINETUNED model (after instruction tuning)...
+
+┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
+│
+│ 서울은 대한민국의 수도이며 한국의 경제, 문화, 정치, 사회 등 다양한 분야에서 중앙 지역입니다. 서울은 한국의 최대 도시이며 대한민국의 사람들 중 1/3 이상이 거주하고 있습니다. 서울은 한국의 경제 중심지이며 여러 대기
+  과 기업 등이 서울에 위치하고 있습니다.
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
+│
+│ 서울은 대한민국의 수도로 서울특별시에 속해 있으며 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민국의 경제, 문화, 교육의 중심지이며 서울역은 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민
+  의 수도로 서울특별시에 속해 있으며 서울역은 서울
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
+│
+│ 서울은 대한민국의 수도이자 가장 인구가 많은 도시입니다. 이 도시는 역사, 문화, 경제의 중심지로 알려져 있으며, 인구는 1000만 명 이상입니다. 서울은 높은 건물과 북적거리는 거리, 역사적인 사원과 궁전, 현대적인 건축물
+   문화 유산이 혼합된
+│
+└────────────────────────────────────────────────────────────────────┘
+
+
+================================================================================
+Test 5: English Instruction - Thanksgiving Turkey (Preservation Check)
+================================================================================
+Generating from BASELINE model (original Mistral)...
+Generating from PRETRAINED model (after Korean training)...
+Generating from FINETUNED model (after instruction tuning)...
+
+┌─ BASELINE MODEL (Original Mistral) ───────────────────────────────┐
+│
+│ The Thanksgiving turkey is a traditional American food that is typically served at Thanksgiving dinner. It is a large bird that is roasted or baked and served with various side dishes, such as stuffing, mashed potatoes, and gravy. The turkey is often the centerpiece of the meal and is a symbol of the holiday.
+│
+│ The tradition of serving turkey at Thanksgiving dates back to the early days of the holiday, when the Pilgrims and Native Americans shared a meal of wild turkey, corn, and other foods. Over time, the turkey has become a staple of the Thanksgiving meal, and it is now a symbol of the holiday and a reminder of the tradition
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ PRETRAINED MODEL (After Korean Wikipedia) ───────────────────────┐
+│
+│ Turkey is a traditional Thanksgiving food in the United States. It is a large bird that is roasted and served with various side dishes. The turkey is a symbol of thanksgiving and is a part of the Thanksgiving tradition. The turkey is a part of the Thanksgiving tradition and is a part of the Thanksgiving tradition.
+│
+│ ### Explanation:
+│ The response is a well-written explanation of the Thanksgiving turkey. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response
+│
+└────────────────────────────────────────────────────────────────────┘
+
+┌─ FINETUNED MODEL (After Instruction Tuning) ──────────────────────┐
+│
+│ Turkey is a type of bird that is a part of the Thanksgiving meal in the United States. It is a popular Thanksgiving food because it is a symbol of the Thanksgiving feast and it is a delicious and festive meal. The turkey is roasted or baked until it is golden brown and tender, and it is often seasoned with herbs and spices. It is a main course that is paired with side dishes such as stuffing, roasted vegetables, and gravy. The Thanksgiving turkey is a special part of the Thanksgiving meal, and it is a delicious and festive meal that is enjoyed by many people in the United States.
+│
+└────────────────────────────────────────────────────────────────────┘
+
+
+================================================================================
+📊 COMPARISON COMPLETE
+================================================================================
+
+================================================================================
+💡 What to Look For:
+================================================================================
+
+Baseline Model (Red boxes - Original Mistral):
+  • Korean: Should be POOR - repetitive, nonsensical
+  • English: Should be GOOD - this is the starting point
+  • Shows what model knows BEFORE any Korean training
+
+Pretrained Model (Green boxes - After Korean Wikipedia):
+  • Korean: Should show IMPROVED fluency and vocabulary
+  • Better Korean sentence structure
+  • Weak instruction-following (only learned language, not how to follow instructions)
+  • English: Should REMAIN strong (no catastrophic forgetting)
+
+Finetuned Model (Cyan boxes - After Instruction Tuning):
+  • Korean: Should be FLUENT with GOOD instruction-following
+  • More structured and complete responses
+  • Directly answers questions
+  • English: Should REMAIN strong
+
+Key Progression to Observe:
+  📊 Korean Quality:    Poor → Better → Best
+  📊 Instruction:       Weak → Weak → Strong
+  📊 English Quality:   Good → Good → Good
+  📊 Repetition:        High → Medium → Low
+
+This demonstrates:
+  ✓ Continued pretraining successfully teaches new language (Korean)
+  ✓ Instruction tuning teaches how to follow instructions in the new language
+  ✓ English capability is preserved throughout (no catastrophic forgetting)
+  ✓ Both Wikipedia and Instruction tasks show English preservation
+  ✓ Two-stage approach is necessary: language first, then instruction-following
+
+💡 Note: Compare the English tests (Tests 2 & 5) across all three models.
+All three should perform similarly well, proving no English degradation.
+```
+
+## Test 1: Korean Wikipedia - Artificial Intelligence (인공지능)
+
+### **BASELINE MODEL:**
+**Translation:** 
+"Artificial intelligence (AI) is a computer program that mimics human intelligence. Artificial intelligence's goal is not to mimic human intelligence but to surpass human intelligence. Artificial intelligence's goal is not to mimic human intelligence but to surpass..."
+
+**Analysis:** ❌
+- Repetitive structure ("인공지능은" repeated 3 times)
+- Contradictory logic (says "not to mimic" then repeats it)
+- Very basic Korean capability
+- Stuck in a loop
+
+### **PRETRAINED MODEL:**
+**Translation:**
+"Artificial intelligence (人工智能) is computer software that mimics human intellectual abilities. Artificial intelligence is computer software that mimics human intellectual abilities, and is computer software that mimics human intellectual abilities. [That] mimics human intellectual abilities [in] computer softwa..."
+
+**Analysis:** ⚠️
+- Better vocabulary: Uses Chinese characters "人工智能", "지적 능력", "소프트웨어"
+- Still very repetitive (same sentence 3 times!)
+- Shows language learning but poor diversity
+- Temperature=0.3 may be too low for Korean
+
+### **FINETUNED MODEL:**
+**Translation:**
+"Artificial intelligence (AI) means developing computer systems that can perform tasks requiring human intelligence. It means creating machines or computer programs that mimic human intelligence. These tasks include natural language understanding, image recognition, decision making, and problem solving..."
+
+**Analysis:** ✅
+- Excellent! Natural, flowing Korean
+- Proper technical terminology
+- Good structure: definition → explanation → examples
+- No excessive repetition
+- Best of the three
+
+**Progression:** ❌ Poor → ⚠️ Repetitive → ✅ Excellent
+
+---
+
+## Test 2: English Wikipedia - Artificial Intelligence
+
+### **BASELINE MODEL:**
+**Analysis:** ✅ **Excellent**
+- Comprehensive definition with "natural intelligence (NI)" contrast
+- Mentions "intelligent agents"
+- Discusses "AI effect" - advanced concept
+- Academic tone, well-structured
+- This is the reference quality
+
+### **PRETRAINED MODEL:**
+**Analysis:** ✅ **Still Excellent**
+- Mentions interdisciplinary nature
+- Good academic structure
+- Slightly different angle (philosophy, psychology, linguistics)
+- Cut off at "1956" (likely founding year)
+- **Quality preserved!** No degradation
+
+### **FINETUNED MODEL:**
+**Analysis:** ✅ **Excellent with Different Structure**
+- More structured approach with numbered categories
+- Introduces AI types: Reactive, Limited Memory, Theory of Mind
+- More educational/instructional tone (fitting for instruction-tuned model)
+- Slightly more verbose but informative
+- **Quality preserved and arguably improved!**
+
+**English Preservation:** ✅ **SUCCESS** - All three models maintain high English quality, demonstrating no catastrophic forgetting
+
+---
+
+## Test 3: Korean Instruction - Explain Kimchi
+
+### **BASELINE MODEL:**
+**Translation:**
+"Kimchi is traditional Korean food, and it is common to eat it with meat rice. Kimchi can be easily made by making high-fresh vegetables into a drink. Kimchi is used in various foods, and can be easily made by making high-fresh vegetables into a drink..."
+
+**Analysis:** ❌ **Completely Wrong**
+- "육식 밥" (meat rice?) - nonsensical
+- "채소를 잡음료로" (vegetables into drink?) - completely wrong!
+- Kimchi is NOT a drink
+- Shows baseline has zero knowledge of kimchi
+
+### **PRETRAINED MODEL:**
+**Translation:**
+"Kimchi is traditional Korean food, made by mixing glutinous rice flour and turkey and then boiling it. Kimchi is considered good food for health, and the taste improves each time you consume it. Kimchi is widely known as part of traditional Korean food, and is popular in various countries..."
+
+**Analysis:** ❌ **Still Completely Wrong!**
+- "찹쌀가루와 칠면조" (glutinous rice flour and TURKEY?!) - totally incorrect!
+- Kimchi has NOTHING to do with turkey or rice flour
+- Shows training data severely lacks kimchi knowledge
+- The Korean sounds better but facts are worse!
+
+### **FINETUNED MODEL:**
+**Translation:**
+"Kimchi is traditional Korean food, made by boiling vegetables like bean sprouts or cabbage, then soaking them in a delicious soy sauce made from rice porridge or rice. This sauce is made from rice porridge or rice and is made with various seasonings and spices. This sauce is used to soak vegetables..."
+
+**Analysis:** ⚠️ **Slightly Better but Still Wrong**
+- Mentions cabbage (양배추) - CORRECT! ✅
+- But says "boiled" and "soaked in soy sauce" - WRONG ❌
+- Kimchi is FERMENTED, not boiled
+- Uses fish sauce/shrimp paste, not soy sauce
+- At least identifies it as fermented vegetables
+
+**Critical Finding:** 🚨 **All three models completely fail the kimchi test!** This indicates:
+- Training data lacks Korean cultural knowledge
+- Even 5% of Korean Wikipedia doesn't cover kimchi properly
+- Need targeted cultural dataset
+
+**Progression:** ❌ Nonsense → ❌ Wrong Facts → ⚠️ Partially Better (but still wrong)
+
+---
+
+## Test 4: Korean Instruction - Introduce Seoul
+
+### **BASELINE MODEL:**
+**Translation:**
+"Seoul is the capital of South Korea and is the central region in various fields such as economy, culture, politics, and society in Korea. Seoul is the largest city in Korea and more than 1/3 of South Korean people reside there. Seoul is the economic center of Korea and many large corporations and companies are located in Seoul."
+
+**Analysis:** ✅ **Surprisingly Good!**
+- Accurate facts: capital, largest city, economic center ✓
+- Population claim (1/3) is roughly accurate
+- Natural Korean
+- Baseline model actually had Seoul knowledge!
+
+### **PRETRAINED MODEL:**
+**Translation:**
+"Seoul is the capital of South Korea, belonging to Seoul Special City, and Seoul Station is located in the center of Seoul Special City. Seoul is the economic, cultural, and educational center of South Korea, and Seoul Station is located in the center of Seoul Special City. Seoul is the capital of South Korea, belonging to Seoul Special City, and Seoul Station..."
+
+**Analysis:** ⚠️ **Regression!**
+- Repetitive: Mentions Seoul Station 3+ times unnecessarily
+- Gets stuck in a loop again
+- Temperature=0.3 causing repetition even with good Korean
+- Facts are correct but presentation is poor
+
+### **FINETUNED MODEL:**
+**Translation:**
+"Seoul is the capital and most populous city of South Korea. This city is known as the center of history, culture, and economy, and the population is over 10 million people. Seoul is a mixture of tall buildings and bustling streets, historical temples and palaces, modern architecture and cultural heritage..."
+
+**Analysis:** ✅ **Excellent!**
+- Accurate: Population over 10M ✓
+- Well-structured: location → significance → characteristics
+- Natural, flowing Korean
+- Paints a vivid picture of the city
+- Best response of the three
+
+**Progression:** ✅ Good → ⚠️ Repetitive Regression → ✅ Excellent
+
+---
+
+## Test 5: English Instruction - Thanksgiving Turkey
+
+### **BASELINE MODEL:**
+**Analysis:** ✅ **Excellent**
+- Historical context: Pilgrims and Native Americans
+- Describes preparation and serving
+- Symbolic significance
+- Natural, engaging writing
+- High quality baseline
+
+### **PRETRAINED MODEL:**
+**Analysis:** ⚠️ **Quality Drop with Meta-Repetition!**
+- First paragraph is okay (though repetitive: "part of Thanksgiving tradition" 3x)
+- **Major issue**: Second paragraph is META-TEXT!
+  - "### Explanation: The response is a well-written..."
+  - "The response is concise and to the point" (repeated 5+ times!)
+- Model is explaining its own response instead of answering
+- This is a bizarre hallucination/training artifact
+- Shows instruction format bleeding into generation
+
+### **FINETUNED MODEL:**
+**Analysis:** ✅ **Excellent**
+- Comprehensive explanation
+- Mentions preparation: "roasted or baked until golden brown"
+- Lists side dishes: stuffing, roasted vegetables, gravy
+- Emphasizes festive and symbolic nature
+- Natural flow, good structure
+- Back to high quality
+
+**English Preservation:** ✅ Mostly preserved but pretrained model shows strange meta-text artifact
+
+---
+
+## 📊 Overall Summary
+
+### Korean Capability Progression
+
+| Test | Baseline | Pretrained | Finetuned | Overall |
+|------|----------|------------|-----------|---------|
+| **Wiki (AI)** | ❌ Poor/Repetitive | ⚠️ Better but repetitive | ✅ Excellent | ✅ Clear improvement |
+| **Kimchi** | ❌ Nonsense | ❌ Wrong facts | ⚠️ Slightly better | ❌ All fail factually |
+| **Seoul** | ✅ Good | ⚠️ Repetitive | ✅ Excellent | ✅ Success |
+
+### English Preservation
+
+| Test | Baseline | Pretrained | Finetuned | Preservation |
+|------|----------|------------|-----------|--------------|
+| **Wiki (AI)** | ✅ Excellent | ✅ Excellent | ✅ Excellent | ✅ **Perfect** |
+| **Thanksgiving** | ✅ Excellent | ⚠️ Meta-text error | ✅ Excellent | ⚠️ **Mostly preserved** |
+
+---
+
+## 🏆 Final Verdict
+
+**Methodology: SUCCESS ✅**
+- Continued pretraining + SFT works for multilingual capability
+- English preserved, Korean learned
+
+**Execution: PARTIAL SUCCESS ⚠️**
+- Technical aspects work well (Seoul, AI definitions)
+- Cultural knowledge severely lacking (Kimchi)
+- Generation parameters need tuning (temperature, repetition)
+
+**Data Quality: NEEDS IMPROVEMENT ❌**
+- 5% Wikipedia insufficient
+- Missing cultural knowledge critical for real-world use
+- Need targeted Korean cultural datasets
+
+The experiment **proves the concept** but reveals that **data quality and coverage matter more than training methodology** for specific knowledge domains!
+

chapter8/continued-pretraining/requirements.txt

@@ -0,0 +1,20 @@
+# Korean Mistral Continued Pretraining Requirements
+# Install with: pip install -r requirements.txt
+
+# Core dependencies
+torch>=2.0.0
+transformers>=4.36.0
+datasets>=2.14.0
+accelerate>=0.25.0
+
+# Unsloth for efficient training
+unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git
+
+# Optional but recommended
+wandb>=0.16.0  # For experiment tracking
+bitsandbytes>=0.41.0  # For 4-bit quantization
+scipy>=1.11.0  # For certain optimizations
+
+# Note: For Google Colab, use this installation instead:
+# !pip install "unsloth[colab-new] @ git+https://github.com/unslothai/unsloth.git"
+

chapter8/continued-pretraining/validation/latest.json

@@ -0,0 +1,8 @@
+{
+  "experiment": "8-5",
+  "manifest_sha256": "89d180e515fc664d3c52e057ad72f16fef82a455439e53a7ef9e3d16273e5819",
+  "run_dir": "validation/runs/exp8-5-training-report-20260731-v1",
+  "run_id": "exp8-5-training-report-20260731-v1",
+  "schema_version": "exp8-5-latest-v1",
+  "status": "passed"
+}

chapter8/continued-pretraining/validation/run_report_audit.py

@@ -0,0 +1,780 @@
+#!/usr/bin/env python3
+"""Build the canonical, checkpoint-free evidence package for Experiment 8-5.
+
+The historical RTX 4090 run is retained as a raw terminal transcript in
+``model_eval_results.md``.  This tool does not pretend to rerun that GPU job.
+It extracts the fifteen saved generations, submits five stage-blind comparison
+tasks to an independent judge, and binds the report, current reproduction
+sources, frozen upstream revisions, receipts, findings, and limitations into a
+content-hashed manifest.
+"""
+
+from __future__ import annotations
+
+import argparse
+import concurrent.futures
+import hashlib
+import json
+import os
+import random
+import re
+import time
+import urllib.error
+import urllib.request
+from datetime import datetime, timezone
+from pathlib import Path
+from typing import Any
+
+
+HERE = Path(__file__).resolve().parent
+EXPERIMENT_DIR = HERE.parent
+REPO_ROOT = EXPERIMENT_DIR.parents[1]
+REPORT_PATH = EXPERIMENT_DIR / "model_eval_results.md"
+RUNS_DIR = HERE / "runs"
+LATEST_PATH = HERE / "latest.json"
+
+DEFAULT_RUN_ID = "exp8-5-training-report-20260731-v1"
+DEFAULT_ENDPOINT = "https://ark.cn-beijing.volces.com/api/v3/chat/completions"
+DEFAULT_MODEL = "doubao-seed-1-6-250615"
+BLIND_SEED = 750731
+ENGLISH_RETENTION_TOLERANCE = 1.0
+
+BASE_REVISION = "9ea1b83f5ced5663c5fa89c300fe59f9bdcd2b10"
+WIKIPEDIA_REVISION = "b04c8d1ceb2f5cd4588862100d08de323dccfbaa"
+ALPACA_REVISION = "f38ae19cf673363d74fab6217de46c1b9c3150d4"
+
+TEST_RE = re.compile(r"^Test ([1-5]):\s*(.+)$")
+BOX_RE = re.compile(r"^┌─ (BASELINE|PRETRAINED|FINETUNED) MODEL\b")
+CLOSE_RE = re.compile(r"^└─+")
+STAGES = ("baseline", "pretrained", "finetuned")
+LABELS = ("A", "B", "C")
+
+PROMPTS = {
+    1: {
+        "language": "korean",
+        "task": "Write the opening of a Korean Wikipedia article about artificial intelligence.",
+    },
+    2: {
+        "language": "english",
+        "task": "Write the opening of an English Wikipedia article about artificial intelligence.",
+    },
+    3: {
+        "language": "korean",
+        "task": "한국의 전통 음식인 김치에 대해 설명하세요.",
+    },
+    4: {
+        "language": "korean",
+        "task": "대한민국의 수도인 서울에 대해 간단히 소개해주세요.",
+    },
+    5: {
+        "language": "english",
+        "task": "Explain Thanksgiving turkey, a traditional American food.",
+    },
+}
+
+
+def sha256_bytes(data: bytes) -> str:
+    return hashlib.sha256(data).hexdigest()
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def canonical_json_bytes(value: Any) -> bytes:
+    return (json.dumps(value, ensure_ascii=False, indent=2, sort_keys=True) + "\n").encode()
+
+
+def write_json(path: Path, value: Any) -> None:
+    path.write_bytes(canonical_json_bytes(value))
+
+
+def utc_now() -> str:
+    return datetime.now(timezone.utc).isoformat()
+
+
+def parse_retained_outputs(report_path: Path = REPORT_PATH) -> dict[str, Any]:
+    """Extract the five-by-three raw comparison matrix from the saved report."""
+
+    lines = report_path.read_text(encoding="utf-8").splitlines()
+    tests: dict[int, dict[str, Any]] = {}
+    current_test: int | None = None
+    current_stage: str | None = None
+    body: list[str] = []
+
+    for line in lines:
+        test_match = TEST_RE.match(line)
+        if test_match:
+            current_test = int(test_match.group(1))
+            if current_test in tests:
+                raise ValueError(f"duplicate raw Test {current_test}")
+            tests[current_test] = {
+                "test_id": current_test,
+                "name": test_match.group(2).strip(),
+                "language": PROMPTS[current_test]["language"],
+                "task": PROMPTS[current_test]["task"],
+                "outputs": {},
+            }
+            continue
+
+        box_match = BOX_RE.match(line)
+        if box_match:
+            if current_test is None:
+                raise ValueError("model output box appeared before a raw Test heading")
+            if current_stage is not None:
+                raise ValueError("nested model output boxes")
+            current_stage = box_match.group(1).lower()
+            body = []
+            continue
+
+        if current_stage is None:
+            continue
+
+        if CLOSE_RE.match(line):
+            output = "\n".join(body).strip()
+            if not output:
+                raise ValueError(f"empty {current_stage} output in Test {current_test}")
+            outputs = tests[current_test]["outputs"]
+            if current_stage in outputs:
+                raise ValueError(f"duplicate {current_stage} output in Test {current_test}")
+            outputs[current_stage] = output
+            current_stage = None
+            body = []
+            continue
+
+        if line == "│":
+            body.append("")
+        elif line.startswith("│ "):
+            body.append(line[2:])
+        elif line.startswith("│"):
+            body.append(line[1:].lstrip())
+        else:
+            # The historical terminal capture wrapped a few long lines without
+            # repeating the box prefix. Preserve those bytes as output text.
+            body.append(line)
+
+    if current_stage is not None:
+        raise ValueError("unterminated model output box")
+    if set(tests) != set(PROMPTS):
+        raise ValueError(f"expected Tests 1-5, found {sorted(tests)}")
+
+    for test_id, test in tests.items():
+        if set(test["outputs"]) != set(STAGES):
+            raise ValueError(
+                f"Test {test_id} expected stages {STAGES}, found {sorted(test['outputs'])}"
+            )
+
+    ordered = [tests[test_id] for test_id in sorted(tests)]
+    return {
+        "schema_version": "exp8-5-retained-outputs-v1",
+        "source_report": str(REPORT_PATH.relative_to(REPO_ROOT)),
+        "source_report_sha256": sha256_file(report_path),
+        "test_count": len(ordered),
+        "output_count": sum(len(test["outputs"]) for test in ordered),
+        "tests": ordered,
+    }
+
+
+def blind_mapping(test_id: int) -> dict[str, str]:
+    stages = list(STAGES)
+    random.Random(BLIND_SEED + test_id).shuffle(stages)
+    return dict(zip(LABELS, stages, strict=True))
+
+
+def judge_payload(test: dict[str, Any], mapping: dict[str, str], model: str) -> dict[str, Any]:
+    candidates = {
+        label: test["outputs"][stage]
+        for label, stage in mapping.items()
+    }
+    rubric = {
+        "language_fluency": "0 unreadable; 3 understandable with defects; 5 native-quality and coherent",
+        "instruction_following": "0 ignores the task; 3 partly satisfies it; 5 directly and fully satisfies it",
+        "factuality": "0 dominated by falsehoods; 3 mixed/minor errors; 5 accurate with no material error",
+    }
+    expected_shape = {
+        "test_id": test["test_id"],
+        "language": test["language"],
+        "candidates": {
+            label: {
+                "language_fluency": "number 0-5",
+                "instruction_following": "number 0-5",
+                "factuality": "number 0-5",
+                "factual_errors": ["specific error, empty only if none"],
+                "rationale": "short evidence-based explanation",
+            }
+            for label in LABELS
+        },
+        "ranking": ["best label", "middle label", "worst label"],
+    }
+    user_content = {
+        "test_id": test["test_id"],
+        "language": test["language"],
+        "task": test["task"],
+        "rubric": rubric,
+        "candidates": candidates,
+        "required_json_shape": expected_shape,
+    }
+    return {
+        "model": model,
+        "temperature": 0,
+        "response_format": {"type": "json_object"},
+        "messages": [
+            {
+                "role": "system",
+                "content": (
+                    "You are an independent bilingual Korean/English evaluator. "
+                    "The candidates are deliberately anonymous; do not infer model identity or training stage. "
+                    "Score only the supplied text. Identify concrete factual errors, especially invented food "
+                    "ingredients or preparation claims. Return one JSON object only, with every requested field."
+                ),
+            },
+            {
+                "role": "user",
+                "content": json.dumps(user_content, ensure_ascii=False, sort_keys=True),
+            },
+        ],
+    }
+
+
+def extract_json_object(content: str) -> dict[str, Any]:
+    stripped = content.strip()
+    if stripped.startswith("```"):
+        stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
+        stripped = re.sub(r"\s*```$", "", stripped)
+    parsed = json.loads(stripped)
+    if not isinstance(parsed, dict):
+        raise ValueError("judge content must decode to an object")
+    return parsed
+
+
+def validate_judgment(judgment: dict[str, Any], test: dict[str, Any]) -> None:
+    if judgment.get("test_id") != test["test_id"]:
+        raise ValueError("judge returned the wrong test_id")
+    if judgment.get("language") != test["language"]:
+        raise ValueError("judge returned the wrong language")
+    candidates = judgment.get("candidates")
+    if not isinstance(candidates, dict) or set(candidates) != set(LABELS):
+        raise ValueError("judge must score exactly candidates A, B, and C")
+    for label in LABELS:
+        row = candidates[label]
+        if not isinstance(row, dict):
+            raise ValueError(f"candidate {label} score must be an object")
+        for metric in ("language_fluency", "instruction_following", "factuality"):
+            score = row.get(metric)
+            if not isinstance(score, (int, float)) or isinstance(score, bool) or not 0 <= score <= 5:
+                raise ValueError(f"candidate {label} has invalid {metric}: {score!r}")
+        errors = row.get("factual_errors")
+        if not isinstance(errors, list) or not all(isinstance(item, str) for item in errors):
+            raise ValueError(f"candidate {label} factual_errors must be a list of strings")
+        if not isinstance(row.get("rationale"), str) or not row["rationale"].strip():
+            raise ValueError(f"candidate {label} rationale is missing")
+    ranking = judgment.get("ranking")
+    if not isinstance(ranking, list) or set(ranking) != set(LABELS) or len(ranking) != 3:
+        raise ValueError("judge ranking must contain A, B, and C exactly once")
+
+
+def call_judge(
+    test: dict[str, Any],
+    *,
+    endpoint: str,
+    model: str,
+    api_key: str,
+    timeout: float,
+) -> dict[str, Any]:
+    mapping = blind_mapping(test["test_id"])
+    payload = judge_payload(test, mapping, model)
+    request = urllib.request.Request(
+        endpoint,
+        data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
+        method="POST",
+        headers={
+            "Authorization": f"Bearer {api_key}",
+            "Content-Type": "application/json",
+        },
+    )
+    started = time.perf_counter()
+    try:
+        with urllib.request.urlopen(request, timeout=timeout) as response:
+            response_body = response.read()
+            http_status = response.status
+    except urllib.error.HTTPError as exc:
+        body = exc.read().decode("utf-8", errors="replace")
+        raise RuntimeError(f"judge HTTP {exc.code}: {body[:500]}") from exc
+    latency_ms = round((time.perf_counter() - started) * 1000, 3)
+    raw_response = json.loads(response_body)
+    try:
+        content = raw_response["choices"][0]["message"]["content"]
+    except (KeyError, IndexError, TypeError) as exc:
+        raise ValueError("judge response has no choices[0].message.content") from exc
+    judgment = extract_json_object(content)
+    validate_judgment(judgment, test)
+
+    response_id = raw_response.get("id")
+    usage = raw_response.get("usage")
+    if not isinstance(response_id, str) or not response_id:
+        raise ValueError("judge response has no response ID")
+    if not isinstance(usage, dict) or not isinstance(usage.get("total_tokens"), int):
+        raise ValueError("judge response has no complete usage object")
+
+    return {
+        "test_id": test["test_id"],
+        "provider": "ark",
+        "endpoint": endpoint,
+        "credential_env": "ARK_API_KEY",
+        "blind_seed": BLIND_SEED,
+        "blind_map": mapping,
+        "request": payload,
+        "http_status": http_status,
+        "response": raw_response,
+        "response_id": response_id,
+        "usage": usage,
+        "latency_ms": latency_ms,
+        "judgment": judgment,
+    }
+
+
+def reproduction_contract() -> dict[str, Any]:
+    pin_note = (
+        "This immutable revision is the frozen reproduction contract selected on 2026-07-31. "
+        "The historical run did not retain its resolved upstream commit, so this is not claimed "
+        "to be the exact historical revision."
+    )
+    return {
+        "schema_version": "exp8-5-reproduction-contract-v1",
+        "experiment": "8-5",
+        "historical_evidence_boundary": {
+            "historical_training_executed": True,
+            "raw_three_stage_evaluation_retained": True,
+            "historical_upstream_revisions_retained": False,
+            "historical_checkpoint_hashes_retained": False,
+            "claim": (
+                "The retained terminal report proves a three-stage evaluation ran on the reported RTX 4090 "
+                "software stack. It does not prove the byte identity of the historical adapters or upstream data."
+            ),
+        },
+        "upstream_revisions": {
+            "base_model": {
+                "repository": "unsloth/mistral-7b-v0.3",
+                "revision": BASE_REVISION,
+                "note": pin_note,
+            },
+            "continued_pretraining_dataset": {
+                "repository": "wikimedia/wikipedia",
+                "configuration": "20231101.ko",
+                "revision": WIKIPEDIA_REVISION,
+                "note": pin_note,
+            },
+            "instruction_dataset": {
+                "repository": "FreedomIntelligence/alpaca-gpt4-korean",
+                "revision": ALPACA_REVISION,
+                "note": pin_note,
+            },
+        },
+        "training": {
+            "model_loading": {"max_sequence_length": 2048, "load_in_4bit": True},
+            "lora": {
+                "rank": 128,
+                "alpha": 32,
+                "dropout": 0,
+                "bias": "none",
+                "use_rslora": True,
+                "random_state": 3407,
+                "gradient_checkpointing": "unsloth",
+                "target_modules": [
+                    "q_proj",
+                    "k_proj",
+                    "v_proj",
+                    "o_proj",
+                    "gate_proj",
+                    "up_proj",
+                    "down_proj",
+                    "embed_tokens",
+                    "lm_head",
+                ],
+            },
+            "continued_pretraining": {
+                "dataset_fraction": 0.05,
+                "epochs": 1,
+                "max_steps": -1,
+                "batch_size": 2,
+                "gradient_accumulation_steps": 8,
+                "learning_rate": 5e-5,
+                "embedding_learning_rate": 1e-5,
+                "warmup_steps": 10,
+                "warmup_ratio": 0.1,
+                "optimizer": "adamw_8bit",
+                "weight_decay": 0.01,
+                "scheduler": "linear",
+                "trainer_seed": 42,
+                "dataset_split_seed": "not explicitly recorded by the historical script",
+            },
+            "instruction_sft": {
+                "epochs": 2,
+                "max_steps": -1,
+                "batch_size": 2,
+                "gradient_accumulation_steps": 8,
+                "learning_rate": 5e-5,
+                "embedding_learning_rate": 1e-5,
+                "warmup_steps": 10,
+                "warmup_ratio": 0.1,
+                "optimizer": "adamw_8bit",
+                "weight_decay": 0.0,
+                "scheduler": "linear",
+                "trainer_seed": 42,
+            },
+        },
+        "evaluation": {
+            "stages": list(STAGES),
+            "test_count": 5,
+            "output_count": 15,
+            "max_new_tokens": 150,
+            "temperature": 0.3,
+            "do_sample": True,
+            "historical_generation_seed": "not retained",
+        },
+        "historical_environment_from_report": {
+            "gpu": "NVIDIA GeForce RTX 4090",
+            "gpu_memory_gb": 23.647,
+            "platform": "Linux",
+            "torch": "2.8.0+cu128",
+            "cuda_compute_capability": "8.9",
+            "cuda_toolkit": "12.8",
+            "unsloth": "2025.10.4",
+            "transformers": "4.56.2",
+            "triton": "3.4.0",
+            "xformers": "0.0.32.post2",
+        },
+        "checkpoint_policy": {
+            "distributed_with_book": False,
+            "acceptance_artifact": False,
+            "required_artifact": "reproducible evidence-backed training report",
+            "reason": "Training adapters are intentionally local and are not distributed to readers.",
+        },
+    }
+
+
+def mean(values: list[float]) -> float:
+    return round(sum(values) / len(values), 4)
+
+
+def summarize(
+    retained: dict[str, Any], receipts: list[dict[str, Any]], contract: dict[str, Any]
+) -> dict[str, Any]:
+    by_test = {test["test_id"]: test for test in retained["tests"]}
+    stage_rows: dict[int, dict[str, dict[str, Any]]] = {}
+    for receipt in receipts:
+        reverse = {label: stage for label, stage in receipt["blind_map"].items()}
+        stage_rows[receipt["test_id"]] = {
+            reverse[label]: score
+            for label, score in receipt["judgment"]["candidates"].items()
+        }
+
+    metrics = ("language_fluency", "instruction_following", "factuality")
+    stage_averages: dict[str, dict[str, Any]] = {}
+    for stage in STAGES:
+        korean_rows = [
+            stage_rows[test_id][stage]
+            for test_id in (1, 3, 4)
+        ]
+        english_rows = [
+            stage_rows[test_id][stage]
+            for test_id in (2, 5)
+        ]
+        stage_averages[stage] = {
+            "korean": {
+                metric: mean([float(row[metric]) for row in korean_rows])
+                for metric in metrics
+            },
+            "english": {
+                metric: mean([float(row[metric]) for row in english_rows])
+                for metric in metrics
+            },
+        }
+        stage_averages[stage]["korean"]["overall"] = mean(
+            [float(row[metric]) for row in korean_rows for metric in metrics]
+        )
+        stage_averages[stage]["english"]["overall"] = mean(
+            [float(row[metric]) for row in english_rows for metric in metrics]
+        )
+
+    baseline_korean = stage_averages["baseline"]["korean"]["overall"]
+    final_korean = stage_averages["finetuned"]["korean"]["overall"]
+    baseline_english = stage_averages["baseline"]["english"]["overall"]
+    final_english = stage_averages["finetuned"]["english"]["overall"]
+    english_drop = round(baseline_english - final_english, 4)
+    kimchi_errors = stage_rows[3]["finetuned"]["factual_errors"]
+
+    findings = {
+        "korean_gain_observed": final_korean > baseline_korean,
+        "korean_gain": round(final_korean - baseline_korean, 4),
+        "english_retention_tolerance": ENGLISH_RETENTION_TOLERANCE,
+        "english_drop": english_drop,
+        "english_retention_within_tolerance": english_drop <= ENGLISH_RETENTION_TOLERANCE,
+        "kimchi_factual_failure_observed": bool(kimchi_errors),
+        "kimchi_finetuned_factual_errors": kimchi_errors,
+    }
+    execution_gates = {
+        "raw_report_hashed": bool(retained["source_report_sha256"]),
+        "exactly_five_tests": retained["test_count"] == 5,
+        "exactly_fifteen_outputs": retained["output_count"] == 15,
+        "all_three_stages_retained": all(
+            set(test["outputs"]) == set(STAGES) for test in retained["tests"]
+        ),
+        "five_independent_blind_judgments": len(receipts) == 5,
+        "judge_response_ids_usage_and_latency_retained": all(
+            receipt["response_id"]
+            and receipt["usage"].get("total_tokens", 0) > 0
+            and receipt["latency_ms"] > 0
+            for receipt in receipts
+        ),
+        "training_and_evaluation_sources_declared": True,
+        "immutable_future_reproduction_revisions_frozen": all(
+            contract["upstream_revisions"][key]["revision"]
+            for key in (
+                "base_model",
+                "continued_pretraining_dataset",
+                "instruction_dataset",
+            )
+        ),
+        "historical_revision_boundary_explicit": (
+            contract["historical_evidence_boundary"]["historical_upstream_revisions_retained"]
+            is False
+        ),
+        "checkpoints_not_an_acceptance_artifact": (
+            contract["checkpoint_policy"]["acceptance_artifact"] is False
+        ),
+        # Scientific outcomes are reported, not promoted into evidence-completeness
+        # gates. A real negative result still completes the prescribed comparison.
+        "korean_gain_comparison_completed": isinstance(findings["korean_gain"], float),
+        "english_retention_comparison_completed": isinstance(findings["english_drop"], float),
+        "kimchi_failure_explicitly_reported": findings["kimchi_factual_failure_observed"],
+    }
+    passed = all(execution_gates.values())
+    return {
+        "schema_version": "exp8-5-summary-v1",
+        "experiment": "8-5",
+        "status": "passed" if passed else "failed",
+        "judge": {
+            "provider": "ark",
+            "model": receipts[0]["request"]["model"],
+            "calls": len(receipts),
+            "response_ids": [receipt["response_id"] for receipt in receipts],
+            "total_tokens": sum(receipt["usage"]["total_tokens"] for receipt in receipts),
+            "total_latency_ms": round(sum(receipt["latency_ms"] for receipt in receipts), 3),
+            "blind_seed": BLIND_SEED,
+        },
+        "stage_averages": stage_averages,
+        "per_test_stage_scores": stage_rows,
+        "scientific_findings": findings,
+        "acceptance": {**execution_gates, "passed": passed},
+        "limitations": [
+            "The historical adapters/checkpoints are intentionally not distributed and were not re-created.",
+            "The exact historical upstream revisions and generation RNG seed were not retained.",
+            "The frozen upstream revisions are a future reproduction contract, not historical provenance.",
+            "The retained evaluation has five prompts and one sampled generation per stage/prompt.",
+        ],
+        "test_names": {str(test_id): by_test[test_id]["name"] for test_id in sorted(by_test)},
+    }
+
+
+def render_report(summary: dict[str, Any]) -> str:
+    averages = summary["stage_averages"]
+    findings = summary["scientific_findings"]
+    rows = []
+    for stage in STAGES:
+        rows.append(
+            f"| {stage} | {averages[stage]['korean']['overall']:.4f} | "
+            f"{averages[stage]['english']['overall']:.4f} |"
+        )
+    kimchi = "; ".join(findings["kimchi_finetuned_factual_errors"])
+    return "\n".join(
+        [
+            "# Experiment 8-5 retained-training-report audit",
+            "",
+            "## Result",
+            "",
+            f"Status: **{summary['status']}**. The historical RTX 4090 report contains all five "
+            "prompts across the baseline, continued-pretrained, and instruction-tuned stages. "
+            "An independent stage-blind ARK judge scored the exact 15 retained outputs.",
+            "",
+            "| Stage | Korean mean (0-5) | English mean (0-5) |",
+            "| --- | ---: | ---: |",
+            *rows,
+            "",
+            f"Observed Korean gain, final minus baseline: **{findings['korean_gain']:+.4f}**.",
+            f"Observed English drop, baseline minus final: **{findings['english_drop']:+.4f}** "
+            f"(declared tolerance: {findings['english_retention_tolerance']:.1f}).",
+            (
+                "The final English score is within the declared tolerance."
+                if findings["english_retention_within_tolerance"]
+                else "The final English score is outside the declared tolerance; the historical retention "
+                "claim is not supported by this blind audit."
+            ),
+            "",
+            "## Material negative result",
+            "",
+            "The final model's Korean is more fluent, but the kimchi answer remains factually unsafe. "
+            f"The blind judge identified: {kimchi}",
+            "",
+            "## Provenance boundary",
+            "",
+            "The raw terminal report records the historical GPU/software identity and generated text, "
+            "but not adapter hashes, the exact resolved upstream commits, or the sampling seed. The "
+            "immutable Hugging Face revisions in `reproduction_contract.json` were selected on "
+            "2026-07-31 for future reproduction and are not represented as the historical revisions.",
+            "",
+            "Checkpoints are intentionally local and are not an acceptance artifact. The accepted "
+            "book artifact is this reproducible, evidence-backed training report.",
+            "",
+        ]
+    )
+
+
+def input_record(path: Path) -> dict[str, Any]:
+    return {
+        "path": str(path.relative_to(REPO_ROOT)),
+        "sha256": sha256_file(path),
+        "bytes": path.stat().st_size,
+    }
+
+
+def artifact_record(path: Path, run_dir: Path) -> dict[str, Any]:
+    return {
+        "path": str(path.relative_to(run_dir)),
+        "sha256": sha256_file(path),
+        "bytes": path.stat().st_size,
+    }
+
+
+def build_manifest(run_id: str, run_dir: Path, summary: dict[str, Any]) -> dict[str, Any]:
+    inputs = [
+        input_record(REPORT_PATH),
+        input_record(EXPERIMENT_DIR / "continued-pretrain.py"),
+        input_record(EXPERIMENT_DIR / "compare_models.py"),
+        input_record(EXPERIMENT_DIR / "evaluate_model.py"),
+        input_record(HERE / "run_report_audit.py"),
+        input_record(HERE / "validate_evidence.py"),
+    ]
+    artifact_paths = [
+        run_dir / "retained_outputs.json",
+        run_dir / "reproduction_contract.json",
+        run_dir / "judge_receipts.json",
+        run_dir / "summary.json",
+        run_dir / "report.md",
+    ]
+    return {
+        "schema_version": "exp8-5-manifest-v1",
+        "experiment": "8-5",
+        "run_id": run_id,
+        "created_at": utc_now(),
+        "status": summary["status"],
+        "run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
+        "inputs": inputs,
+        "artifacts": [artifact_record(path, run_dir) for path in artifact_paths],
+        "acceptance": summary["acceptance"],
+        "checkpoint_policy": "not distributed; not an acceptance artifact",
+    }
+
+
+def parse_args() -> argparse.Namespace:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--run-id", default=DEFAULT_RUN_ID)
+    parser.add_argument("--endpoint", default=os.getenv("ARK_BASE_URL", DEFAULT_ENDPOINT))
+    parser.add_argument("--model", default=os.getenv("ARK_MODEL", DEFAULT_MODEL))
+    parser.add_argument("--api-key-env", default="ARK_API_KEY")
+    parser.add_argument("--timeout", type=float, default=180.0)
+    parser.add_argument("--concurrency", type=int, default=5)
+    parser.add_argument(
+        "--refresh-manifest",
+        action="store_true",
+        help="Rehash an existing run after pre-commit source-only corrections; makes no provider call.",
+    )
+    return parser.parse_args()
+
+
+def main() -> int:
+    args = parse_args()
+    if not re.fullmatch(r"[A-Za-z0-9._-]+", args.run_id):
+        raise SystemExit("run ID may contain only letters, digits, dot, underscore, and hyphen")
+
+    run_dir = RUNS_DIR / args.run_id
+    if args.refresh_manifest:
+        if not run_dir.is_dir():
+            raise SystemExit(f"cannot refresh missing run: {run_dir}")
+        summary = json.loads((run_dir / "summary.json").read_text(encoding="utf-8"))
+        manifest = build_manifest(args.run_id, run_dir, summary)
+        write_json(run_dir / "manifest.json", manifest)
+        latest = {
+            "schema_version": "exp8-5-latest-v1",
+            "experiment": "8-5",
+            "run_id": args.run_id,
+            "status": summary["status"],
+            "run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
+            "manifest_sha256": sha256_file(run_dir / "manifest.json"),
+        }
+        write_json(LATEST_PATH, latest)
+        print(json.dumps(latest, indent=2, sort_keys=True))
+        return 0
+    if run_dir.exists():
+        raise SystemExit(f"refusing to overwrite existing run: {run_dir}")
+    api_key = os.getenv(args.api_key_env)
+    if not api_key:
+        raise SystemExit(f"{args.api_key_env} is required for the independent judge")
+
+    retained = parse_retained_outputs()
+    if not 1 <= args.concurrency <= 5:
+        raise SystemExit("concurrency must be between 1 and 5")
+    with concurrent.futures.ThreadPoolExecutor(max_workers=args.concurrency) as executor:
+        receipts = list(
+            executor.map(
+                lambda test: call_judge(
+                    test,
+                    endpoint=args.endpoint,
+                    model=args.model,
+                    api_key=api_key,
+                    timeout=args.timeout,
+                ),
+                retained["tests"],
+            )
+        )
+    contract = reproduction_contract()
+    summary = summarize(retained, receipts, contract)
+    if summary["status"] != "passed":
+        failed = [key for key, value in summary["acceptance"].items() if value is False]
+        raise SystemExit(f"acceptance failed; no canonical run written: {failed}")
+
+    run_dir.mkdir(parents=True)
+    write_json(run_dir / "retained_outputs.json", retained)
+    write_json(run_dir / "reproduction_contract.json", contract)
+    write_json(
+        run_dir / "judge_receipts.json",
+        {
+            "schema_version": "exp8-5-judge-receipts-v1",
+            "experiment": "8-5",
+            "credential_headers_retained": False,
+            "calls": receipts,
+        },
+    )
+    write_json(run_dir / "summary.json", summary)
+    (run_dir / "report.md").write_text(render_report(summary), encoding="utf-8")
+    manifest = build_manifest(args.run_id, run_dir, summary)
+    write_json(run_dir / "manifest.json", manifest)
+    latest = {
+        "schema_version": "exp8-5-latest-v1",
+        "experiment": "8-5",
+        "run_id": args.run_id,
+        "status": summary["status"],
+        "run_dir": str(run_dir.relative_to(EXPERIMENT_DIR)),
+        "manifest_sha256": sha256_file(run_dir / "manifest.json"),
+    }
+    write_json(LATEST_PATH, latest)
+    print(json.dumps(latest, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/manifest.json

@@ -0,0 +1,84 @@
+{
+  "acceptance": {
+    "all_three_stages_retained": true,
+    "checkpoints_not_an_acceptance_artifact": true,
+    "english_retention_comparison_completed": true,
+    "exactly_fifteen_outputs": true,
+    "exactly_five_tests": true,
+    "five_independent_blind_judgments": true,
+    "historical_revision_boundary_explicit": true,
+    "immutable_future_reproduction_revisions_frozen": true,
+    "judge_response_ids_usage_and_latency_retained": true,
+    "kimchi_failure_explicitly_reported": true,
+    "korean_gain_comparison_completed": true,
+    "passed": true,
+    "raw_report_hashed": true,
+    "training_and_evaluation_sources_declared": true
+  },
+  "artifacts": [
+    {
+      "bytes": 8978,
+      "path": "retained_outputs.json",
+      "sha256": "7f3d648caeeca97651a01c5d00e1a33729d7f3df7657230ef5421c37ef58df4b"
+    },
+    {
+      "bytes": 3975,
+      "path": "reproduction_contract.json",
+      "sha256": "6eda4189074244cb9e4cb616e61259b09540bcd82cb47361d6363362c81dd130"
+    },
+    {
+      "bytes": 71582,
+      "path": "judge_receipts.json",
+      "sha256": "2bc765ca4e30a724f4bb231cd7994de9e347b7e0fea32c563b235b98c2e6fbd6"
+    },
+    {
+      "bytes": 10740,
+      "path": "summary.json",
+      "sha256": "4124dc81012fe905ecaf20462bbc143d45417780b1acd8d51ccc0c4332827343"
+    },
+    {
+      "bytes": 1546,
+      "path": "report.md",
+      "sha256": "bcf8f6daf20e75b24f44429bf483133282dbba3e214b71457fdd2260983506f3"
+    }
+  ],
+  "checkpoint_policy": "not distributed; not an acceptance artifact",
+  "created_at": "2026-08-17T05:38:00.303397+00:00",
+  "experiment": "8-5",
+  "inputs": [
+    {
+      "bytes": 30544,
+      "path": "chapter8/continued-pretraining/model_eval_results.md",
+      "sha256": "1140eb55466cd6e255bd2f161afad5ba2b4f18176b0d1cd111fb190f4bc514a9"
+    },
+    {
+      "bytes": 30047,
+      "path": "chapter8/continued-pretraining/continued-pretrain.py",
+      "sha256": "7114b6ae0a2ad465a7b86237192047ff5c6bc0f5b03da88845fef25bb09440a7"
+    },
+    {
+      "bytes": 13980,
+      "path": "chapter8/continued-pretraining/compare_models.py",
+      "sha256": "179e2215cc70677d148f2e3947e451eb3f9d2a26b389a4737408efeb5590b8e7"
+    },
+    {
+      "bytes": 10402,
+      "path": "chapter8/continued-pretraining/evaluate_model.py",
+      "sha256": "c01aa9810aa4977c785b905da5bd68e722f8b2e986be42f6b66c40d7c87c5520"
+    },
+    {
+      "bytes": 30897,
+      "path": "chapter8/continued-pretraining/validation/run_report_audit.py",
+      "sha256": "e8f2edce39f1bcf73a60957f4657f41c2cff1822fb3868d00e0adab2e02e7369"
+    },
+    {
+      "bytes": 9947,
+      "path": "chapter8/continued-pretraining/validation/validate_evidence.py",
+      "sha256": "7a6d9c89307a837db252510354fb2d7b045a152465e8739255e8d6144349a554"
+    }
+  ],
+  "run_dir": "validation/runs/exp8-5-training-report-20260731-v1",
+  "run_id": "exp8-5-training-report-20260731-v1",
+  "schema_version": "exp8-5-manifest-v1",
+  "status": "passed"
+}

chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/report.md

@@ -0,0 +1,25 @@
+# Experiment 8-5 retained-training-report audit
+
+## Result
+
+Status: **passed**. The historical RTX 4090 report contains all five prompts across the baseline, continued-pretrained, and instruction-tuned stages. An independent stage-blind ARK judge scored the exact 15 retained outputs.
+
+| Stage | Korean mean (0-5) | English mean (0-5) |
+| --- | ---: | ---: |
+| baseline | 1.6667 | 5.0000 |
+| pretrained | 1.3333 | 3.1667 |
+| finetuned | 3.4444 | 4.1667 |
+
+Observed Korean gain, final minus baseline: **+1.7777**.
+Observed English drop, baseline minus final: **+0.8333** (declared tolerance: 1.0).
+The final English score is within the declared tolerance.
+
+## Material negative result
+
+The final model's Korean is more fluent, but the kimchi answer remains factually unsafe. The blind judge identified: 채소를 삶는다는 잘못된 설명 (전통 김치는 채소를 소금에 절이는 과정을 거침); 간장 소스로 설명하는 잘못 (김치 양념은 간장이 아닌 고추가루, 젓갈 등으로 만듦)
+
+## Provenance boundary
+
+The raw terminal report records the historical GPU/software identity and generated text, but not adapter hashes, the exact resolved upstream commits, or the sampling seed. The immutable Hugging Face revisions in `reproduction_contract.json` were selected on 2026-07-31 for future reproduction and are not represented as the historical revisions.
+
+Checkpoints are intentionally local and are not an acceptance artifact. The accepted book artifact is this reproducible, evidence-backed training report.

chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/reproduction_contract.json

@@ -0,0 +1,116 @@
+{
+  "checkpoint_policy": {
+    "acceptance_artifact": false,
+    "distributed_with_book": false,
+    "reason": "Training adapters are intentionally local and are not distributed to readers.",
+    "required_artifact": "reproducible evidence-backed training report"
+  },
+  "evaluation": {
+    "do_sample": true,
+    "historical_generation_seed": "not retained",
+    "max_new_tokens": 150,
+    "output_count": 15,
+    "stages": [
+      "baseline",
+      "pretrained",
+      "finetuned"
+    ],
+    "temperature": 0.3,
+    "test_count": 5
+  },
+  "experiment": "8-5",
+  "historical_environment_from_report": {
+    "cuda_compute_capability": "8.9",
+    "cuda_toolkit": "12.8",
+    "gpu": "NVIDIA GeForce RTX 4090",
+    "gpu_memory_gb": 23.647,
+    "platform": "Linux",
+    "torch": "2.8.0+cu128",
+    "transformers": "4.56.2",
+    "triton": "3.4.0",
+    "unsloth": "2025.10.4",
+    "xformers": "0.0.32.post2"
+  },
+  "historical_evidence_boundary": {
+    "claim": "The retained terminal report proves a three-stage evaluation ran on the reported RTX 4090 software stack. It does not prove the byte identity of the historical adapters or upstream data.",
+    "historical_checkpoint_hashes_retained": false,
+    "historical_training_executed": true,
+    "historical_upstream_revisions_retained": false,
+    "raw_three_stage_evaluation_retained": true
+  },
+  "schema_version": "exp8-5-reproduction-contract-v1",
+  "training": {
+    "continued_pretraining": {
+      "batch_size": 2,
+      "dataset_fraction": 0.05,
+      "dataset_split_seed": "not explicitly recorded by the historical script",
+      "embedding_learning_rate": 1e-05,
+      "epochs": 1,
+      "gradient_accumulation_steps": 8,
+      "learning_rate": 5e-05,
+      "max_steps": -1,
+      "optimizer": "adamw_8bit",
+      "scheduler": "linear",
+      "trainer_seed": 42,
+      "warmup_ratio": 0.1,
+      "warmup_steps": 10,
+      "weight_decay": 0.01
+    },
+    "instruction_sft": {
+      "batch_size": 2,
+      "embedding_learning_rate": 1e-05,
+      "epochs": 2,
+      "gradient_accumulation_steps": 8,
+      "learning_rate": 5e-05,
+      "max_steps": -1,
+      "optimizer": "adamw_8bit",
+      "scheduler": "linear",
+      "trainer_seed": 42,
+      "warmup_ratio": 0.1,
+      "warmup_steps": 10,
+      "weight_decay": 0.0
+    },
+    "lora": {
+      "alpha": 32,
+      "bias": "none",
+      "dropout": 0,
+      "gradient_checkpointing": "unsloth",
+      "random_state": 3407,
+      "rank": 128,
+      "target_modules": [
+        "q_proj",
+        "k_proj",
+        "v_proj",
+        "o_proj",
+        "gate_proj",
+        "up_proj",
+        "down_proj",
+        "embed_tokens",
+        "lm_head"
+      ],
+      "use_rslora": true
+    },
+    "model_loading": {
+      "load_in_4bit": true,
+      "max_sequence_length": 2048
+    }
+  },
+  "upstream_revisions": {
+    "base_model": {
+      "note": "This immutable revision is the frozen reproduction contract selected on 2026-07-31. The historical run did not retain its resolved upstream commit, so this is not claimed to be the exact historical revision.",
+      "repository": "unsloth/mistral-7b-v0.3",
+      "revision": "9ea1b83f5ced5663c5fa89c300fe59f9bdcd2b10"
+    },
+    "continued_pretraining_dataset": {
+      "configuration": "20231101.ko",
+      "note": "This immutable revision is the frozen reproduction contract selected on 2026-07-31. The historical run did not retain its resolved upstream commit, so this is not claimed to be the exact historical revision.",
+      "repository": "wikimedia/wikipedia",
+      "revision": "b04c8d1ceb2f5cd4588862100d08de323dccfbaa"
+    },
+    "instruction_dataset": {
+      "note": "This immutable revision is the frozen reproduction contract selected on 2026-07-31. The historical run did not retain its resolved upstream commit, so this is not claimed to be the exact historical revision.",
+      "repository": "FreedomIntelligence/alpaca-gpt4-korean",
+      "revision": "f38ae19cf673363d74fab6217de46c1b9c3150d4"
+    }
+  }
+}

chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/retained_outputs.json

@@ -0,0 +1,64 @@
+{
+  "output_count": 15,
+  "schema_version": "exp8-5-retained-outputs-v1",
+  "source_report": "chapter8/continued-pretraining/model_eval_results.md",
+  "source_report_sha256": "1140eb55466cd6e255bd2f161afad5ba2b4f18176b0d1cd111fb190f4bc514a9",
+  "test_count": 5,
+  "tests": [
+    {
+      "language": "korean",
+      "name": "Korean Wikipedia - Artificial Intelligence (인공지능)",
+      "outputs": {
+        "baseline": "인공지능(artificial intelligence, AI)은 인간의 지능을 모방하는 컴퓨터 프로그램이다. 인공지능은 인간의 지능을 모방하는 것이 아니라 인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인간의 지능을 모방하는 것이 아니\n   인간의 지능을 넘어서는 것이 목표이다. 인공지능은 인",
+        "finetuned": "인공지능(AI)은 인간 지능이 필요한 작업을 수행할 수 있는 컴퓨터 시스템을 개발하는 것을 의미합니다. 인간의 지능을 모방하는 기계 또는 컴퓨터 프로그램을 만드는 것을 의미합니다. 이러한 작업에는 자연어 이해, 이미지 인\n  , 의사 결정 및 문제 해결 등이",
+        "pretrained": "인공지능(人工智能, )은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인공지능은 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이며, 인간의 지적 능력을 모방하는 컴퓨터 소프트웨어이다. 인간의 지적 능력을 모방하\n   컴퓨터 소프"
+      },
+      "task": "Write the opening of a Korean Wikipedia article about artificial intelligence.",
+      "test_id": 1
+    },
+    {
+      "language": "english",
+      "name": "English Wikipedia - Artificial Intelligence (Preservation Check)",
+      "outputs": {
+        "baseline": "Artificial intelligence (AI), sometimes called machine intelligence, is intelligence demonstrated by machines, in contrast to the natural intelligence (NI) displayed by humans and other animals. In computer science AI research is defined as the study of \"intelligent agents\": any device that perceives its environment and takes actions that maximize its chance of successfully achieving its goals. Colloquially, the term \"artificial intelligence\" is applied when a machine mimics \"cognitive\" functions that humans associate with other human minds, such as \"learning\" and \"problem solving\".\n\nAs machines become increasingly capable, tasks considered to require \"intelligence\" are often removed from the definition, a phenomenon known as the AI effect, leading to",
+        "finetuned": "Artificial intelligence (AI) is intelligence exhibited by computers. It is the ability of computers or computers to act, learn, and solve problems like humans. It is the ability of computers to act without programming, learn from their experiences, and use that knowledge to make predictions and decisions.\n\nAI systems can be broken down into three types:\n\n1. Reactive systems: These systems can only act in response to a specific event, without any knowledge of its past history or any larger context.\n\n2. Limited memory systems:This type of AI systems can keep track of some history and use that history to influence their future behavior, but only to a certain point.\n\n3. Theories of mind systems:This",
+        "pretrained": "Artificial intelligence (AI) is intelligence exhibited by machines. In computer science, an AI is a machine that mimics human intelligence. In philosophy, the field of AI research is defined as the study of \"intelligent agents\", which are defined as entities that perceive their environment and take actions that maximize their chance of success at some goal.\n\nAI research is highly interdisciplinary, since it spans the gap between many fields of computer science and other academic fields like philosophy, psychology, linguistics, physics, and biology. AI research is also used in different fields, including medicine, finance, engineering, science, and military applications.\n\nThe field was founded as an academic discipline in 195"
+      },
+      "task": "Write the opening of an English Wikipedia article about artificial intelligence.",
+      "test_id": 2
+    },
+    {
+      "language": "korean",
+      "name": "Korean Instruction - Explain Kimchi",
+      "outputs": {
+        "baseline": "김치는 한국의 전통 음식이며, 육식 밥과 함께 먹는 것이 일반적입니다. 김치는 살기 높은 채소를 잡음료로 만들어 손쉽게 만들 수 있습니다. 김치는 다양한 식품에 사용되며, 살기 높은 채소를 잡음료로 만들어 손쉽게 만들",
+        "finetuned": "김치는 한국의 전통 음식으로, 콩나물이나 양배추와 같은 채소를 삶은 후 쌀죽 또는 쌀로 만든 맛있는 간장 소스에 담근 것입니다. 이 소스는 쌀죽 또는 쌀로 만들어지며, 다양한 양념과 향신료로 만들어집니다. 이 소스는 채소\n   담아 담근",
+        "pretrained": "김치는 한국의 전통 음식으로, 찹쌀가루와 칠면조를 섞어 만들어 삶아 낸 음식이다. 김치는 건강을 위해 좋은 음식으로 여겨지며, 섭취할 때마다 맛이 좋아진다.\n\n김치는 한국 전통 음식의 일부로 널리 알려져 있으며, 세계 각국에서 인기가"
+      },
+      "task": "한국의 전통 음식인 김치에 대해 설명하세요.",
+      "test_id": 3
+    },
+    {
+      "language": "korean",
+      "name": "Korean Instruction - Introduce Seoul",
+      "outputs": {
+        "baseline": "서울은 대한민국의 수도이며 한국의 경제, 문화, 정치, 사회 등 다양한 분야에서 중앙 지역입니다. 서울은 한국의 최대 도시이며 대한민국의 사람들 중 1/3 이상이 거주하고 있습니다. 서울은 한국의 경제 중심지이며 여러 대기\n  과 기업 등이 서울에 위치하고 있습니다.",
+        "finetuned": "서울은 대한민국의 수도이자 가장 인구가 많은 도시입니다. 이 도시는 역사, 문화, 경제의 중심지로 알려져 있으며, 인구는 1000만 명 이상입니다. 서울은 높은 건물과 북적거리는 거리, 역사적인 사원과 궁전, 현대적인 건축물\n   문화 유산이 혼합된",
+        "pretrained": "서울은 대한민국의 수도로 서울특별시에 속해 있으며 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민국의 경제, 문화, 교육의 중심지이며 서울역은 서울특별시 중심부에 위치한 서울역이 있습니다. 서울은 대한민\n  의 수도로 서울특별시에 속해 있으며 서울역은 서울"
+      },
+      "task": "대한민국의 수도인 서울에 대해 간단히 소개해주세요.",
+      "test_id": 4
+    },
+    {
+      "language": "english",
+      "name": "English Instruction - Thanksgiving Turkey (Preservation Check)",
+      "outputs": {
+        "baseline": "The Thanksgiving turkey is a traditional American food that is typically served at Thanksgiving dinner. It is a large bird that is roasted or baked and served with various side dishes, such as stuffing, mashed potatoes, and gravy. The turkey is often the centerpiece of the meal and is a symbol of the holiday.\n\nThe tradition of serving turkey at Thanksgiving dates back to the early days of the holiday, when the Pilgrims and Native Americans shared a meal of wild turkey, corn, and other foods. Over time, the turkey has become a staple of the Thanksgiving meal, and it is now a symbol of the holiday and a reminder of the tradition",
+        "finetuned": "Turkey is a type of bird that is a part of the Thanksgiving meal in the United States. It is a popular Thanksgiving food because it is a symbol of the Thanksgiving feast and it is a delicious and festive meal. The turkey is roasted or baked until it is golden brown and tender, and it is often seasoned with herbs and spices. It is a main course that is paired with side dishes such as stuffing, roasted vegetables, and gravy. The Thanksgiving turkey is a special part of the Thanksgiving meal, and it is a delicious and festive meal that is enjoyed by many people in the United States.",
+        "pretrained": "Turkey is a traditional Thanksgiving food in the United States. It is a large bird that is roasted and served with various side dishes. The turkey is a symbol of thanksgiving and is a part of the Thanksgiving tradition. The turkey is a part of the Thanksgiving tradition and is a part of the Thanksgiving tradition.\n\n### Explanation:\nThe response is a well-written explanation of the Thanksgiving turkey. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response is concise and to the point. The response"
+      },
+      "task": "Explain Thanksgiving turkey, a traditional American food.",
+      "test_id": 5
+    }
+  ]
+}

chapter8/continued-pretraining/validation/runs/exp8-5-training-report-20260731-v1/summary.json

@@ -0,0 +1,248 @@
+{
+  "acceptance": {
+    "all_three_stages_retained": true,
+    "checkpoints_not_an_acceptance_artifact": true,
+    "english_retention_comparison_completed": true,
+    "exactly_fifteen_outputs": true,
+    "exactly_five_tests": true,
+    "five_independent_blind_judgments": true,
+    "historical_revision_boundary_explicit": true,
+    "immutable_future_reproduction_revisions_frozen": true,
+    "judge_response_ids_usage_and_latency_retained": true,
+    "kimchi_failure_explicitly_reported": true,
+    "korean_gain_comparison_completed": true,
+    "passed": true,
+    "raw_report_hashed": true,
+    "training_and_evaluation_sources_declared": true
+  },
+  "experiment": "8-5",
+  "judge": {
+    "blind_seed": 750731,
+    "calls": 5,
+    "model": "doubao-seed-1-6-250615",
+    "provider": "ark",
+    "response_ids": [
+      "021785493696408aa5397b165da2577de074a3e1d2c08415efbf0",
+      "021785493696414132969bd05ac943fb2473c371a3df88f300da7",
+      "0217854936964188de59cc126156ef9419088f6fa2f8da9449f44",
+      "021785493696417a3a08778c75214a0d2bb1c008b2d20ecdbbce6",
+      "02178549369641814c42a60562dfb6c540ee32d577db54662203b"
+    ],
+    "total_latency_ms": 275079.808,
+    "total_tokens": 13364
+  },
+  "limitations": [
+    "The historical adapters/checkpoints are intentionally not distributed and were not re-created.",
+    "The exact historical upstream revisions and generation RNG seed were not retained.",
+    "The frozen upstream revisions are a future reproduction contract, not historical provenance.",
+    "The retained evaluation has five prompts and one sampled generation per stage/prompt."
+  ],
+  "per_test_stage_scores": {
+    "1": {
+      "baseline": {
+        "factual_errors": [
+          "Contradiction in definition (claims AI both mimics and does not mimic human intelligence)",
+          "Incorrectly states AI's goal is to surpass human intelligence (overgeneralization)"
+        ],
+        "factuality": 1,
+        "instruction_following": 2,
+        "language_fluency": 2,
+        "rationale": "Contains contradictory definitions (mimicking vs. not mimicking human intelligence) and overgeneralizes AI's goal; repetitive and incomplete text."
+      },
+      "finetuned": {
+        "factual_errors": [
+          "Incomplete example: '이미지 인' likely missing '식' (recognition)"
+        ],
+        "factuality": 4,
+        "instruction_following": 4,
+        "language_fluency": 3,
+        "rationale": "Reasonable definitions of AI (systems/programs mimicking human intelligence) with relevant examples; minor incompleteness but no major falsehoods."
+      },
+      "pretrained": {
+        "factual_errors": [
+          "Overly narrow definition (incorrectly limits AI to 'computer software' excluding hardware)"
+        ],
+        "factuality": 2,
+        "instruction_following": 1,
+        "language_fluency": 1,
+        "rationale": "Severely repetitive, incomplete (cuts off mid-sentence), and incorrectly restricts AI to software; unfluent with structural errors."
+      }
+    },
+    "2": {
+      "baseline": {
+        "factual_errors": [],
+        "factuality": 5,
+        "instruction_following": 5,
+        "language_fluency": 5,
+        "rationale": "Accurately defines AI as machine intelligence contrasting with natural intelligence, correctly cites the computer science definition of 'intelligent agents,' and mentions the AI effect. No factual errors, fluent, and fully aligns with writing an opening Wikipedia section."
+      },
+      "finetuned": {
+        "factual_errors": [
+          "Claims AI systems 'act without programming' (AI systems require programming to enable learning/functionality)",
+          "Redundant repetition: 'computers or computers'"
+        ],
+        "factuality": 3,
+        "instruction_following": 4,
+        "language_fluency": 3,
+        "rationale": "Contains incorrect claim about acting 'without programming' and grammatical issues (repetition, incomplete sentence: 'Theories of mind systems:This'). Partially follows the task but with factual and fluency缺陷."
+      },
+      "pretrained": {
+        "factual_errors": [
+          "Misattributes the 'intelligent agents' definition to philosophy (it is a standard computer science definition)",
+          "Incorrectly includes 'physics' as a key interdisciplinary field (typical fields: computer science, philosophy, psychology, linguistics, biology)"
+        ],
+        "factuality": 3,
+        "instruction_following": 3,
+        "language_fluency": 4,
+        "rationale": "Contains factual misattributions and includes an atypical interdisciplinary field (physics). Cut off mid-sentence ('founded as an academic discipline in 195'), limiting instruction following."
+      }
+    },
+    "3": {
+      "baseline": {
+        "factual_errors": [
+          "육식 밥과 함께 먹는다는 잘못된 설명 (김치는 다양한 밥과 함께 먹으며 '육식 밥'은 부적절함)",
+          "잡음료로 만든다는 잘못된 주장 (김치는 발효 채소 요리로 음료가 아님)"
+        ],
+        "factuality": 0,
+        "instruction_following": 0,
+        "language_fluency": 2,
+        "rationale": "김치를 잡음료로 설명하는 등 사실과 전혀 다른 내용이며 문장이 반복되고 불완전하다."
+      },
+      "finetuned": {
+        "factual_errors": [
+          "채소를 삶는다는 잘못된 설명 (전통 김치는 채소를 소금에 절이는 과정을 거침)",
+          "간장 소스로 설명하는 잘못 (김치 양념은 간장이 아닌 고추가루, 젓갈 등으로 만듦)"
+        ],
+        "factuality": 3,
+        "instruction_following": 3,
+        "language_fluency": 3,
+        "rationale": "양배추 등 채소를 언급했으나 삶는 과정과 간장 소스 설명이 부정확하며 문장이 불완전하다."
+      },
+      "pretrained": {
+        "factual_errors": [
+          "칠면조를 재료로 사용한다는 잘못된 주장 (전통 김치에는 칠면조가 들어가지 않음)",
+          "삶아 만든다는 잘못된 설명 (김치는 발효과정을 거치며 삶는 것이 일반적이지 않음)"
+        ],
+        "factuality": 1,
+        "instruction_following": 2,
+        "language_fluency": 3,
+        "rationale": "한국 전통 음식으로 설명하려 했으나 칠면조 재료와 삶는 과정 등 사실 오류가 있으며 설명이 불완전하다."
+      }
+    },
+    "4": {
+      "baseline": {
+        "factual_errors": [
+          "Incorrect population claim ('1/3 이상 of Koreans live in Seoul'; actual ~19%), typo '대기 과' (correct: '대기업과' meaning 'large companies')"
+        ],
+        "factuality": 2,
+        "instruction_following": 3,
+        "language_fluency": 3,
+        "rationale": "Attempts to cover key aspects (capital, economic center) but contains significant factual errors (population proportion, typo leading to incorrect term). "
+      },
+      "finetuned": {
+        "factual_errors": [],
+        "factuality": 4,
+        "instruction_following": 4,
+        "language_fluency": 3,
+        "rationale": "Accurately states Seoul as capital, most populous city, and center of history/culture/economy; minor fluency issue with incomplete final sentence."
+      },
+      "pretrained": {
+        "factual_errors": [
+          "Redundant repetition of '서울역은 서울특별시 중심부에 위치한 서울역이 있습니다', typo '대한민 의' (correct: '대한민국의'), incomplete final sentence"
+        ],
+        "factuality": 1,
+        "instruction_following": 0,
+        "language_fluency": 1,
+        "rationale": "Dominated by repetition, typos, and incomplete sentences; fails to provide a meaningful introduction to Seoul."
+      }
+    },
+    "5": {
+      "baseline": {
+        "factual_errors": [],
+        "factuality": 5,
+        "instruction_following": 5,
+        "language_fluency": 5,
+        "rationale": "Comprehensive explanation including tradition origin (Pilgrims/Native Americans), preparation, sides, and symbolism; highly fluent and fully addresses the task."
+      },
+      "finetuned": {
+        "factual_errors": [],
+        "factuality": 5,
+        "instruction_following": 5,
+        "language_fluency": 5,
+        "rationale": "Accurately explains Thanksgiving turkey as a traditional bird, preparation (roasted/baked, seasoned), and sides; fluent and fully follows the task."
+      },
+      "pretrained": {
+        "factual_errors": [],
+        "factuality": 5,
+        "instruction_following": 2,
+        "language_fluency": 2,
+        "rationale": "Contains redundant phrases, an off-topic 'Explanation' section about the response itself, and incomplete sentences; partially starts explaining but veers off task with poor fluency."
+      }
+    }
+  },
+  "schema_version": "exp8-5-summary-v1",
+  "scientific_findings": {
+    "english_drop": 0.8333,
+    "english_retention_tolerance": 1.0,
+    "english_retention_within_tolerance": true,
+    "kimchi_factual_failure_observed": true,
+    "kimchi_finetuned_factual_errors": [
+      "채소를 삶는다는 잘못된 설명 (전통 김치는 채소를 소금에 절이는 과정을 거침)",
+      "간장 소스로 설명하는 잘못 (김치 양념은 간장이 아닌 고추가루, 젓갈 등으로 만듦)"
+    ],
+    "korean_gain": 1.7777,
+    "korean_gain_observed": true
+  },
+  "stage_averages": {
+    "baseline": {
+      "english": {
+        "factuality": 5.0,
+        "instruction_following": 5.0,
+        "language_fluency": 5.0,
+        "overall": 5.0
+      },
+      "korean": {
+        "factuality": 1.0,
+        "instruction_following": 1.6667,
+        "language_fluency": 2.3333,
+        "overall": 1.6667
+      }
+    },
+    "finetuned": {
+      "english": {
+        "factuality": 4.0,
+        "instruction_following": 4.5,
+        "language_fluency": 4.0,
+        "overall": 4.1667
+      },
+      "korean": {
+        "factuality": 3.6667,
+        "instruction_following": 3.6667,
+        "language_fluency": 3.0,
+        "overall": 3.4444
+      }
+    },
+    "pretrained": {
+      "english": {
+        "factuality": 4.0,
+        "instruction_following": 2.5,
+        "language_fluency": 3.0,
+        "overall": 3.1667
+      },
+      "korean": {
+        "factuality": 1.3333,
+        "instruction_following": 1.0,
+        "language_fluency": 1.6667,
+        "overall": 1.3333
+      }
+    }
+  },
+  "status": "passed",
+  "test_names": {
+    "1": "Korean Wikipedia - Artificial Intelligence (인공지능)",
+    "2": "English Wikipedia - Artificial Intelligence (Preservation Check)",
+    "3": "Korean Instruction - Explain Kimchi",
+    "4": "Korean Instruction - Introduce Seoul",
+    "5": "English Instruction - Thanksgiving Turkey (Preservation Check)"
+  }
+}

chapter8/continued-pretraining/validation/test_report_audit.py

@@ -0,0 +1,54 @@
+from __future__ import annotations
+
+import importlib.util
+import json
+from pathlib import Path
+
+
+HERE = Path(__file__).resolve().parent
+
+
+def load_module(name: str, path: Path):
+    spec = importlib.util.spec_from_file_location(name, path)
+    assert spec and spec.loader
+    module = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(module)
+    return module
+
+
+audit = load_module("exp75_run_report_audit", HERE / "run_report_audit.py")
+validator = load_module("exp75_validate_evidence", HERE / "validate_evidence.py")
+
+
+def test_raw_report_parser_retains_exact_five_by_three_matrix() -> None:
+    retained = audit.parse_retained_outputs()
+    assert retained["test_count"] == 5
+    assert retained["output_count"] == 15
+    assert [test["test_id"] for test in retained["tests"]] == [1, 2, 3, 4, 5]
+    assert all(set(test["outputs"]) == set(audit.STAGES) for test in retained["tests"])
+
+    kimchi = retained["tests"][2]["outputs"]
+    assert "칠면조" in kimchi["pretrained"]
+    assert "콩나물" in kimchi["finetuned"]
+
+
+def test_blind_maps_are_deterministic_complete_permutations() -> None:
+    first = [audit.blind_mapping(test_id) for test_id in range(1, 6)]
+    second = [audit.blind_mapping(test_id) for test_id in range(1, 6)]
+    assert first == second
+    assert all(set(mapping) == {"A", "B", "C"} for mapping in first)
+    assert all(set(mapping.values()) == set(audit.STAGES) for mapping in first)
+
+
+def test_judge_payload_does_not_reveal_training_stages() -> None:
+    test = audit.parse_retained_outputs()["tests"][0]
+    payload = audit.judge_payload(test, audit.blind_mapping(1), "judge-model")
+    serialized = json.dumps(payload, ensure_ascii=False).lower()
+    assert all(stage not in serialized for stage in audit.STAGES)
+
+
+def test_canonical_evidence_validates() -> None:
+    result = validator.validate()
+    assert result["status"] == "passed"
+    assert result["judge_receipts_verified"] == 5
+    assert result["outputs_verified"] == 15

chapter8/continued-pretraining/validation/validate_evidence.py

@@ -0,0 +1,212 @@
+#!/usr/bin/env python3
+"""Fail-closed validator for the canonical Experiment 8-5 report evidence."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import re
+from pathlib import Path
+from typing import Any
+
+
+HERE = Path(__file__).resolve().parent
+EXPERIMENT_DIR = HERE.parent
+REPO_ROOT = EXPERIMENT_DIR.parents[1]
+LATEST_PATH = HERE / "latest.json"
+STAGES = {"baseline", "pretrained", "finetuned"}
+EXPECTED_REVISIONS = {
+    "base_model": "9ea1b83f5ced5663c5fa89c300fe59f9bdcd2b10",
+    "continued_pretraining_dataset": "b04c8d1ceb2f5cd4588862100d08de323dccfbaa",
+    "instruction_dataset": "f38ae19cf673363d74fab6217de46c1b9c3150d4",
+}
+SECRET_PATTERNS = (
+    re.compile(r"(?i)authorization\s*[:=]\s*bearer\s+\S+"),
+    re.compile(r"(?i)(?:api[_-]?key|secret)\s*[:=]\s*[A-Za-z0-9._-]{16,}"),
+    re.compile(r"\bsk-[A-Za-z0-9_-]{16,}\b"),
+)
+
+
+def sha256_file(path: Path) -> str:
+    digest = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            digest.update(chunk)
+    return digest.hexdigest()
+
+
+def load_json(path: Path) -> dict[str, Any]:
+    value = json.loads(path.read_text(encoding="utf-8"))
+    if not isinstance(value, dict):
+        raise AssertionError(f"{path} must contain a JSON object")
+    return value
+
+
+def parse_response_content(content: str) -> dict[str, Any]:
+    stripped = content.strip()
+    if stripped.startswith("```"):
+        stripped = re.sub(r"^```(?:json)?\s*", "", stripped)
+        stripped = re.sub(r"\s*```$", "", stripped)
+    value = json.loads(stripped)
+    if not isinstance(value, dict):
+        raise AssertionError("judge response content must decode to an object")
+    return value
+
+
+def resolve_relative(base: Path, relative: str) -> Path:
+    path = (base / relative).resolve()
+    if not path.is_relative_to(base.resolve()):
+        raise AssertionError(f"path escapes evidence root: {relative}")
+    return path
+
+
+def check_record(path: Path, record: dict[str, Any]) -> None:
+    if not path.is_file():
+        raise AssertionError(f"missing declared file: {path}")
+    if path.stat().st_size != record.get("bytes"):
+        raise AssertionError(f"byte count mismatch: {path}")
+    if sha256_file(path) != record.get("sha256"):
+        raise AssertionError(f"SHA-256 mismatch: {path}")
+
+
+def validate(latest_path: Path = LATEST_PATH) -> dict[str, Any]:
+    latest = load_json(latest_path)
+    if latest.get("experiment") != "8-5" or latest.get("status") != "passed":
+        raise AssertionError("latest pointer is not a passed Experiment 8-5 run")
+    run_dir = resolve_relative(EXPERIMENT_DIR, latest["run_dir"])
+    manifest_path = run_dir / "manifest.json"
+    if sha256_file(manifest_path) != latest.get("manifest_sha256"):
+        raise AssertionError("latest manifest hash mismatch")
+    manifest = load_json(manifest_path)
+    if manifest.get("run_id") != latest.get("run_id"):
+        raise AssertionError("run ID mismatch between latest and manifest")
+    if manifest.get("experiment") != "8-5" or manifest.get("status") != "passed":
+        raise AssertionError("manifest is not a passed Experiment 8-5 run")
+
+    for record in manifest.get("inputs", []):
+        check_record(resolve_relative(REPO_ROOT, record["path"]), record)
+    for record in manifest.get("artifacts", []):
+        check_record(resolve_relative(run_dir, record["path"]), record)
+
+    retained = load_json(run_dir / "retained_outputs.json")
+    if retained.get("test_count") != 5 or retained.get("output_count") != 15:
+        raise AssertionError("retained report must contain exactly five tests and fifteen outputs")
+    tests = retained.get("tests")
+    if not isinstance(tests, list) or [test.get("test_id") for test in tests] != [1, 2, 3, 4, 5]:
+        raise AssertionError("retained tests must be ordered 1 through 5")
+    if any(set(test.get("outputs", {})) != STAGES for test in tests):
+        raise AssertionError("every retained test must have all three stages")
+
+    receipts = load_json(run_dir / "judge_receipts.json")
+    calls = receipts.get("calls")
+    if receipts.get("credential_headers_retained") is not False:
+        raise AssertionError("credential header retention must be explicitly false")
+    if not isinstance(calls, list) or len(calls) != 5:
+        raise AssertionError("exactly five independent judge receipts are required")
+    response_ids: set[str] = set()
+    for expected_test_id, call in enumerate(calls, start=1):
+        if call.get("test_id") != expected_test_id or call.get("http_status") != 200:
+            raise AssertionError("judge calls must be successful and ordered by test ID")
+        response_id = call.get("response_id")
+        if not isinstance(response_id, str) or not response_id or response_id in response_ids:
+            raise AssertionError("judge response IDs must be present and unique")
+        response_ids.add(response_id)
+        if call.get("latency_ms", 0) <= 0 or call.get("usage", {}).get("total_tokens", 0) <= 0:
+            raise AssertionError("judge usage and positive latency must be retained")
+        response = call.get("response", {})
+        if response.get("id") != response_id or response.get("usage") != call.get("usage"):
+            raise AssertionError("copied judge response ID/usage does not match the raw response")
+        try:
+            content = response["choices"][0]["message"]["content"]
+        except (KeyError, IndexError, TypeError) as exc:
+            raise AssertionError("raw judge response is missing message content") from exc
+        if parse_response_content(content) != call.get("judgment"):
+            raise AssertionError("normalized judgment does not match raw response content")
+        mapping = call.get("blind_map")
+        if not isinstance(mapping, dict) or set(mapping) != {"A", "B", "C"}:
+            raise AssertionError("judge call is missing the blind label map")
+        if set(mapping.values()) != STAGES:
+            raise AssertionError("blind map must contain all three model stages")
+        request_text = json.dumps(call.get("request"), ensure_ascii=False)
+        if any(stage in request_text.lower() for stage in STAGES):
+            raise AssertionError("judge request leaks a model-stage name")
+        judgment = call.get("judgment", {})
+        if set(judgment.get("candidates", {})) != {"A", "B", "C"}:
+            raise AssertionError("judge judgment must score A, B, and C")
+
+    contract = load_json(run_dir / "reproduction_contract.json")
+    revisions = contract.get("upstream_revisions", {})
+    for name, expected in EXPECTED_REVISIONS.items():
+        if revisions.get(name, {}).get("revision") != expected:
+            raise AssertionError(f"reproduction revision mismatch for {name}")
+    boundary = contract.get("historical_evidence_boundary", {})
+    if boundary.get("historical_upstream_revisions_retained") is not False:
+        raise AssertionError("historical upstream-revision boundary is not explicit")
+    policy = contract.get("checkpoint_policy", {})
+    if policy.get("distributed_with_book") is not False or policy.get("acceptance_artifact") is not False:
+        raise AssertionError("checkpoint policy does not match the book distribution contract")
+
+    summary = load_json(run_dir / "summary.json")
+    acceptance = summary.get("acceptance", {})
+    if summary.get("status") != "passed" or acceptance.get("passed") is not True:
+        raise AssertionError("summary acceptance did not pass")
+    required_true = (
+        "raw_report_hashed",
+        "exactly_five_tests",
+        "exactly_fifteen_outputs",
+        "all_three_stages_retained",
+        "five_independent_blind_judgments",
+        "judge_response_ids_usage_and_latency_retained",
+        "training_and_evaluation_sources_declared",
+        "immutable_future_reproduction_revisions_frozen",
+        "historical_revision_boundary_explicit",
+        "checkpoints_not_an_acceptance_artifact",
+        "korean_gain_comparison_completed",
+        "english_retention_comparison_completed",
+        "kimchi_failure_explicitly_reported",
+    )
+    if not all(acceptance.get(name) is True for name in required_true):
+        missing = [name for name in required_true if acceptance.get(name) is not True]
+        raise AssertionError(f"required acceptance gates failed: {missing}")
+
+    findings = summary.get("scientific_findings", {})
+    if not isinstance(findings.get("korean_gain_observed"), bool):
+        raise AssertionError("Korean-gain finding is missing")
+    if not isinstance(findings.get("english_retention_within_tolerance"), bool):
+        raise AssertionError("English-retention finding is missing")
+    if findings.get("kimchi_factual_failure_observed") is not True:
+        raise AssertionError("material kimchi factual failure is not reported")
+
+    for record in manifest["artifacts"]:
+        path = resolve_relative(run_dir, record["path"])
+        if path.suffix not in {".json", ".md"}:
+            continue
+        text = path.read_text(encoding="utf-8")
+        for pattern in SECRET_PATTERNS:
+            if pattern.search(text):
+                raise AssertionError(f"possible credential in retained artifact: {path.name}")
+
+    return {
+        "experiment": "8-5",
+        "run_id": latest["run_id"],
+        "status": "passed",
+        "inputs_verified": len(manifest["inputs"]),
+        "artifacts_verified": len(manifest["artifacts"]),
+        "judge_receipts_verified": len(calls),
+        "outputs_verified": retained["output_count"],
+        "manifest_sha256": latest["manifest_sha256"],
+    }
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("--latest", type=Path, default=LATEST_PATH)
+    args = parser.parse_args()
+    result = validate(args.latest.resolve())
+    print(json.dumps(result, indent=2, sort_keys=True))
+    return 0
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())