ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter1/learning-from-experience/README.md

@@ -0,0 +1,755 @@
+# Learning from Experience: RL vs LLM In-Context Learning / 从经验中学习：RL 与 LLM 上下文学习对比
+
+> Compares tabular Q-learning with LLM in-context learning on a treasure-hunt game with hidden mechanics (Shunyu Yao, “The Second Half”).  
+> 代码位于第 1 章项目树；对应书中 **实验 7-1 ★（Q-learning 在寻宝游戏中的表现）** 与 **实验 7-2 ★★（传统 RL 与 LLM Agent 的对比研究）**。
+
+← [Chapter 1 index / 返回第 1 章目录](../README.md) · 📖 [Read Chapter 7 / 读第 7 章正文](../../book/chapter7.md)（[EN](../../book-en/chapter7.md)）
+
+---
+
+## English
+
+### Overview
+
+This experiment compares traditional Reinforcement Learning (Q-learning) with LLM-based in-context learning, replicating the key insights from Shunyu Yao's blog post ["The Second Half"](https://ysymyth.github.io/The-Second-Half/).
+
+It demonstrates how LLMs can generalize through reasoning while traditional RL methods require extensive training to learn game mechanics. We use a text-based treasure hunt game with hidden mechanics that agents must discover through experience.
+
+### Key Insights Being Tested
+
+1. **Sample Efficiency**: LLMs can learn from far fewer examples than traditional RL
+2. **Generalization**: LLMs use reasoning to understand patterns, while RL memorizes state-action mappings
+3. **Prior Knowledge**: Language pre-training provides powerful priors for reasoning about new tasks
+4. **Hidden Mechanics Discovery**: LLMs can form hypotheses and test them, while RL requires exhaustive exploration
+
+### What You'll See
+
+When running the LLM experiment, you'll see the **complete decision-making process**:
+
+```
+============================================================
+LLM DECISION PROCESS
+============================================================
+📊 Experiences in memory: 15
+🎮 Current room: hallway
+🎯 Available actions: 8
+
+💡 Recent successful patterns learned:
+   • take red key → +5.0 reward
+   • try crafting → +10.0 reward
+
+🤔 LLM is thinking...
+
+📝 LLM Reasoning:
+----------------------------------------
+  Based on my past experiences, I've learned that:
+  1. The red key opens the locked door to the guard room
+  2. Crafting rusty sword + magic crystal creates a silver sword
+  3. The silver sword can defeat the strong guard
+  
+  Since I have the silver sword and I'm in the hallway...
+----------------------------------------
+
+✅ Chosen action: go north
+```
+
+This transparency shows exactly how the LLM learns and reasons, unlike the black-box nature of Q-learning.
+
+### The Game
+
+A text-based treasure hunt game where agents must:
+
+- Navigate through multiple rooms
+- Collect items and keys
+- Defeat guards using appropriate weapons
+- Discover hidden mechanics through experience
+
+#### Hidden Mechanics (Not Revealed to Agents)
+
+1. **Color-coded locks**: Specific colored keys open matching doors
+2. **Weapon effectiveness**: Different weapons work against different enemies
+3. **Crafting system**: Certain items combine to create better items
+4. **Potion effects**: Temporary abilities from consuming potions
+
+### Quick Start
+
+#### Installation
+
+```bash
+# Recommended from the repository root: use the shared Chapter 1 environment
+uv sync --locked --extra ch1
+
+# Activate it before changing directories:
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+# pip fallback when uv is not installed:
+# python -m pip install -e ".[ch1]"
+
+# Enter this experiment directory for the commands below
+cd chapter1/learning-from-experience
+
+# Single-project compatibility path, still supported during migration:
+# python -m pip install -r requirements.txt
+```
+
+- Q-learning runs fully offline with **no API key**.
+- The LLM path needs a Moonshot/Kimi API key (or OpenRouter fallback).
+
+#### Setting up Kimi K3 API
+
+To run the LLM experiments, you need a Kimi (Moonshot) API key:
+
+1. Get your API key from [Moonshot AI](https://platform.moonshot.cn/)
+2. Set the environment variable:
+
+```bash
+export LLM_PROVIDER="moonshot"  # or dashscope/qwen/bailian
+export MOONSHOT_API_KEY="your-api-key-here"
+# For Alibaba Cloud Model Studio / Bailian (Qwen), use:
+# export LLM_PROVIDER="dashscope"
+# export DASHSCOPE_API_KEY="your-dashscope-api-key-here"
+# export DASHSCOPE_MODEL="qwen3.7-plus"
+```
+
+Or create a `.env` file:
+
+```bash
+echo "MOONSHOT_API_KEY=your-api-key-here" > .env
+```
+
+**Universal OpenRouter fallback**: if `MOONSHOT_API_KEY` is unset but `OPENROUTER_API_KEY` is set, the LLM path routes through OpenRouter. Because Kimi models are not stably available on OpenRouter, the fallback uses `OPENROUTER_MODEL` (default `openai/gpt-5.6-luna`):
+
+```bash
+export OPENROUTER_API_KEY=your-openrouter-api-key
+python quick_demo.py   # runs via OpenRouter when MOONSHOT_API_KEY is missing
+```
+
+### Running the Experiment
+
+#### Quick Demo (See LLM Learning in Action)
+
+```bash
+python quick_demo.py
+```
+
+This shows a detailed view of how the LLM learns through reasoning, displaying:
+
+- Complete thought process for each decision
+- How experiences accumulate and influence future decisions
+- The dramatic difference in learning speed vs traditional RL
+
+#### Command-Line Interface (`experiment.py`)
+
+`experiment.py` provides a full CLI (Chinese help text). List all flags:
+
+```bash
+python experiment.py --help
+```
+
+Main parameters:
+
+| Parameter | Description | Default |
+| --- | --- | --- |
+| `--mode {both,qlearning,rl,llm}` | Which agent(s): `qlearning`/`rl` = Q-learning only (offline), `llm` = LLM Agent only, `both` = comparison | `both` |
+| `--rl-episodes` | Q-learning training episodes (Experiment 7-1 uses 10000) | `10000` |
+| `--llm-episodes` | LLM Agent training episodes | `20` |
+| `--eval-episodes` | Greedy evaluation episodes after Q-learning | `100` |
+| `--checkpoint-interval` | Learning-curve sample interval (every N episodes) | `1000` |
+| `--model` | LLM model name (or `MOONSHOT_MODEL` env) | `kimi-k3` |
+| `--output` | Results output directory | `results` |
+| `--seed` | Random seed for reproducible Q-learning curves | unset |
+| `--learning-rate` / `--discount` / `--epsilon-decay` / `--epsilon-min` | Q-learning hyperparameters | `0.2 / 0.99 / 0.9995 / 0.1` |
+| `--stochastic` | Use stochastic environment | deterministic |
+| `--skip-llm` | Legacy alias for `--mode qlearning` | — |
+
+#### Q-Learning Only (Experiment 7-1, offline, no API)
+
+```bash
+python experiment.py --mode qlearning --rl-episodes 10000 --seed 42
+```
+
+Training finishes in under ~3 seconds and prints a **learning curve table** showing how the agent goes from ~0% win rate to mastery over nearly 10k episodes (see Results below).
+
+#### Full Comparison (RL vs LLM, Experiment 7-2)
+
+```bash
+python experiment.py --mode both --model kimi-k3
+```
+
+For the exact book protocol and acceptance-grade evidence, use the canonical
+runner. It executes the 10,000-episode Q-learning arm, 100 greedy evaluation
+episodes, and exactly one first-attempt official Moonshot Kimi K3 trajectory:
+
+```bash
+python run_experiment_7_2.py
+```
+
+The canonical runner rejects OpenRouter substitution, API errors, missing raw
+provider response IDs/content, and any parser fallback. It writes
+`validation/<timestamp>/evidence.json`; if only post-run serialization needs to
+be repaired, `finalize_experiment_7_2.py <campaign-dir>` finalizes the already
+saved raw campaign without repeating paid model calls.
+
+This will:
+
+1. Train a Q-learning agent for 10000 episodes (~3 seconds) and print its learning curve
+2. Train an LLM agent for 20 episodes with detailed reasoning display
+3. Evaluate both agents
+4. Generate comparison plots
+5. Save results to the `results/` directory
+
+**Note**: `experiment.py` is the exploratory multi-episode runner. The canonical
+book campaign is deliberately one first attempt. The accepted 2026-07-30 Kimi
+K3 attempt took 416.11 seconds for 17 sequential reasoning calls; the earlier
+“1–2 minutes per game” estimate was not reproduced on this route.
+
+#### LLM Only
+
+```bash
+python experiment.py --mode llm --llm-episodes 20
+```
+
+#### Interactive Game Play
+
+Test the game manually:
+
+```python
+from game_environment import TreasureHuntGame
+
+game = TreasureHuntGame()
+print(game.get_state_description())
+print("Available actions:", game.get_available_actions())
+
+# Try an action
+feedback, reward, done = game.execute_action("take rusty sword")
+print(f"Feedback: {feedback}")
+print(f"Reward: {reward}")
+```
+
+### Validation
+
+Install the `dev` extra from the repository root before running pytest in a clean environment:
+
+```bash
+uv sync --locked --extra ch1 --extra dev
+
+# Activate it before changing directories:
+# macOS/Linux:
+source .venv/bin/activate
+# Windows PowerShell: .venv\Scripts\Activate.ps1
+# Windows cmd: .venv\Scripts\activate.bat
+
+cd chapter1/learning-from-experience
+python -m pytest tests
+```
+
+The longer Q-learning learning-curve check is an offline manual smoke script, kept out of default pytest discovery:
+
+```bash
+python tests/manual/rl_learning_check.py --episodes 1000
+```
+
+### Experiment Results
+
+#### Metrics Compared
+
+1. **Sample Efficiency** — Episodes needed to achieve good performance; learning speed
+2. **Performance** — Victory rate in evaluation; average rewards and episode lengths
+3. **Computational Cost** — Training time; memory (Q-table size vs. experience storage); API calls for LLM
+
+#### Visualizations
+
+The experiment creates comparison plots showing:
+
+- Learning curves over time
+- Victory rate progression
+- Sample efficiency comparison
+- Key insights summary
+
+#### Expected Results
+
+##### Q-Learning learning curve (measured locally, `--mode qlearning --rl-episodes 10000 --seed 42`)
+
+Measured curve (deterministic env; victory rate as a sliding window over the last 1000 episodes; full training ~3s):
+
+| Episodes | Victory rate | Q-table states | epsilon |
+| ---: | ---: | ---: | ---: |
+| 1000 | 0.3% | 123 | 0.606 |
+| 2000 | 0.0% | 123 | 0.368 |
+| 3000 | 0.1% | 126 | 0.223 |
+| 5000 | 0.1% | 128 | 0.100 |
+| 7000 | 97.0% | 138 | 0.100 |
+| 8000 | 99.6% | 138 | 0.100 |
+| 9000 | 99.8% | 139 | 0.100 |
+| 10000 | **98.1%** | 142 | 0.100 |
+
+After training, the canonical greedy evaluation reaches **100%** win rate,
+averaging 12 steps. The accepted Kimi K3 arm won on its first attempt in 17
+steps with 17/17 real responses, zero API errors, zero fallbacks, and 28,242
+tokens. This reproduces the first-attempt conclusion but not the manuscript's
+historical point estimates of exactly 18 Kimi steps and an 11-step Q-learning
+solution. See the [canonical evidence](validation/20260730_011704/evidence.json).
+
+##### RL vs LLM (Experiment 7-2 conclusions)
+
+- **Q-Learning**: Needs ~10000 episodes for stable clears; treats “door / key / sword” as meaningless symbols and only explores statistically.
+- **LLM In-Context**: Carries pretrained priors; often clears in the first episode within tens of steps by reasoning about game concepts.
+- **Sample Efficiency**: LLM is 2–3 orders of magnitude more sample-efficient; but per-episode inference is slow (~1–2 min API), while Q-learning finishes 10000 episodes in ~3s—trade-off depends on interaction cost (see book Experiment 7-2).
+
+### Project Structure
+
+```
+learning-from-experience/
+├── game_environment.py    # Text-based game with hidden mechanics
+├── rl_agent.py            # Q-learning implementation
+├── llm_agent.py           # LLM with in-context learning
+├── experiment.py          # Main experiment runner
+├── demo.py                # Interactive local game demo
+├── quick_demo.py          # Short LLM learning demo
+├── run_experiment_7_2.py  # Exact real campaign + acceptance gates
+├── finalize_experiment_7_2.py # Evidence-only recovery; no API rerun
+├── env.example            # Optional API-key template
+├── tests/
+│   ├── test_basic.py
+│   ├── test_zero_episodes.py
+│   ├── test_rl_progress_small_episodes.py
+│   └── manual/
+│       └── rl_learning_check.py
+├── requirements.txt       # Python dependencies
+├── README.md              # This file
+└── results/               # Experiment outputs (created on run)
+    └── [timestamp]/
+        ├── rl_agent.pkl           # Trained Q-learning agent
+        ├── llm_experiences.json   # LLM's collected experiences
+        ├── experiment_results.json # Numerical results
+        └── comparison_plots.png   # Visualization
+```
+
+### Technical Details
+
+#### Q-Learning Agent
+
+- **Algorithm**: Tabular Q-learning with ε-greedy exploration
+- **State Representation**: Hashed combination of room, inventory, and game state
+- **Learning Rate**: 0.2 (configurable via `--learning-rate`)
+- **Discount Factor**: 0.99 (configurable via `--discount`)
+- **Exploration**: ε starts at 1.0, decays by `--epsilon-decay` (0.9995) to `--epsilon-min` (0.1)
+
+#### LLM Agent (Kimi K3)
+
+- **Model**: `kimi-k3` (override with `--model` or `MOONSHOT_MODEL`)
+- **Reasoning model**: Kimi K3 emits a chain-of-thought (`message.reasoning_content`) before its final answer (`message.content`), so the code uses a generous `max_tokens=2048` to make sure the `ACTION:` line is not truncated by the reasoning budget.
+- **Learning Method**: In-context learning with experience memory (up to 50 experiences)
+- **Context Management**: Stores successful and failed experiences
+- **Reasoning**: Prompts LLM to reason about past experiences before acting
+- **Temperature**: requested 0.7, but reasoning models (Kimi K3, GPT-5) only accept `temperature=1`, so the code auto-forces `1` for those (see `_reasoning_safe_temperature`)
+
+### Extending the Experiment
+
+#### Ideas for Further Research
+
+1. **Different Games**: Try other hidden-mechanic games
+2. **Hybrid Approaches**: Combine RL with LLM guidance
+3. **Transfer Learning**: Test how well agents transfer to similar games
+4. **Ablation Studies**: Remove reasoning prompts to isolate their impact
+5. **Other LLMs**: Compare different language models
+
+#### Modifying the Game
+
+Edit `game_environment.py` to:
+
+- Add new rooms and items
+- Create more complex hidden mechanics
+- Adjust difficulty and rewards
+- Add new types of puzzles
+
+### Educational Value
+
+1. **The Power of Priors**: How language pre-training provides useful knowledge
+2. **Reasoning vs. Memorization**: Different approaches to learning
+3. **Sample Efficiency**: Why it matters for real-world applications
+4. **The Second Half Thesis**: Moving from “can we solve it?” to “how efficiently?”
+
+### References
+
+- [The Second Half](https://ysymyth.github.io/The-Second-Half/) by Shunyu Yao
+- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629)
+- Original Q-learning paper: Watkins & Dayan (1992)
+
+---
+
+## 中文
+
+### 概述
+
+本实验对比传统强化学习（Q-learning）与基于 LLM 的上下文学习（in-context learning），复现 Shunyu Yao 博客 [“The Second Half”](https://ysymyth.github.io/The-Second-Half/) 中的核心洞见。
+
+目标：展示 LLM 如何通过**推理**泛化，而传统 RL 往往需要大量试错才能学到游戏机制。我们使用带有**隐藏机制**的文本寻宝游戏，智能体只能通过经验去发现规则。
+
+代码在 `chapter1/learning-from-experience/`，对应书中**实验 7-1** 与 **实验 7-2**（正文见第 7 章）。
+
+### 要验证的关键洞察
+
+1. **样本效率**：LLM 用远少于传统 RL 的样例即可学习  
+2. **泛化**：LLM 用推理理解模式；RL 记忆状态-动作映射  
+3. **先验知识**：语言预训练为新任务推理提供强大先验  
+4. **隐藏机制发现**：LLM 可形成假设并检验；RL 往往需要穷尽式探索  
+
+### 你会看到什么
+
+运行 LLM 实验时，会看到**完整决策过程**：
+
+```
+============================================================
+LLM DECISION PROCESS
+============================================================
+📊 Experiences in memory: 15
+🎮 Current room: hallway
+🎯 Available actions: 8
+
+💡 Recent successful patterns learned:
+   • take red key → +5.0 reward
+   • try crafting → +10.0 reward
+
+🤔 LLM is thinking...
+
+📝 LLM Reasoning:
+----------------------------------------
+  Based on my past experiences, I've learned that:
+  1. The red key opens the locked door to the guard room
+  2. Crafting rusty sword + magic crystal creates a silver sword
+  3. The silver sword can defeat the strong guard
+  
+  Since I have the silver sword and I'm in the hallway...
+----------------------------------------
+
+✅ Chosen action: go north
+```
+
+这种透明度展示了 LLM 如何学习与推理，有别于 Q-learning 的黑盒性质。
+
+### 游戏说明
+
+文本寻宝游戏，智能体需要：
+
+- 在多个房间间导航  
+- 收集物品与钥匙  
+- 使用合适武器击败守卫  
+- 通过经验发现隐藏机制  
+
+#### 隐藏机制（不对智能体公开）
+
+1. **颜色锁**：特定颜色钥匙开对应门  
+2. **武器有效性**：不同武器对不同敌人有效  
+3. **合成系统**：特定物品可合成更强物品  
+4. **药水效果**：消耗药水获得临时能力  
+
+### 快速开始
+
+#### 安装
+
+```bash
+# 推荐在仓库根目录使用统一的第 1 章环境
+uv sync --locked --extra ch1
+
+# 切换目录前先激活环境：
+# macOS/Linux：
+source .venv/bin/activate
+# Windows PowerShell：.venv\Scripts\Activate.ps1
+# Windows cmd：.venv\Scripts\activate.bat
+
+# 未安装 uv 时可用 pip 兜底：
+# python -m pip install -e ".[ch1]"
+
+# 进入本实验目录，后续命令都在这里运行
+cd chapter1/learning-from-experience
+
+# 迁移期间仍支持单项目兼容路径：
+# python -m pip install -r requirements.txt
+```
+
+- Q-learning 完全离线，**无需任何 API Key**。  
+- LLM 部分需要 Moonshot/Kimi API Key（或 OpenRouter 兜底）。  
+
+#### 配置 Kimi K3 API
+
+运行 LLM 实验需要 Kimi（Moonshot）API Key：
+
+1. 从 [Moonshot AI](https://platform.moonshot.cn/) 获取 Key  
+2. 设置环境变量：
+
+```bash
+export MOONSHOT_API_KEY="your-api-key-here"
+```
+
+或创建 `.env`：
+
+```bash
+echo "MOONSHOT_API_KEY=your-api-key-here" > .env
+```
+
+**通用兜底（OpenRouter）**：若未设置 `MOONSHOT_API_KEY` 但设置了 `OPENROUTER_API_KEY`，LLM 部分会自动改走 OpenRouter。由于 Kimi 模型在 OpenRouter 上不稳定可用，兜底时会使用 `OPENROUTER_MODEL`（默认 `openai/gpt-5.6-luna`）：
+
+```bash
+export OPENROUTER_API_KEY=your-openrouter-api-key
+python quick_demo.py   # MOONSHOT_API_KEY 缺失时自动经 OpenRouter 运行
+```
+
+### 运行实验
+
+#### 快速演示（观察 LLM 如何学习）
+
+```bash
+python quick_demo.py
+```
+
+会展示：
+
+- 每一步的完整思考过程  
+- 经验如何累积并影响后续决策  
+- 与传统 RL 在学习速度上的巨大差异  
+
+#### 命令行接口（`experiment.py`）
+
+`experiment.py` 提供带中文帮助的完整 CLI：
+
+```bash
+python experiment.py --help
+```
+
+主要参数：
+
+| 参数 | 说明 | 默认值 |
+| --- | --- | --- |
+| `--mode {both,qlearning,rl,llm}` | 运行哪种智能体：`qlearning`/`rl` 只跑 Q-learning（离线）、`llm` 只跑 LLM Agent、`both` 两者对比 | `both` |
+| `--rl-episodes` | Q-learning 训练局数（实验 7-1 用 10000） | `10000` |
+| `--llm-episodes` | LLM Agent 训练局数 | `20` |
+| `--eval-episodes` | Q-learning 训练后贪婪评估局数 | `100` |
+| `--checkpoint-interval` | 学习曲线采样间隔（每 N 局记录一次胜率/Q 表规模） | `1000` |
+| `--model` | LLM 模型名（也可用 `MOONSHOT_MODEL` 环境变量） | `kimi-k3` |
+| `--output` | 结果输出目录 | `results` |
+| `--seed` | 随机种子，用于复现 Q-learning 学习曲线 | 不固定 |
+| `--learning-rate` / `--discount` / `--epsilon-decay` / `--epsilon-min` | Q-learning 超参数 | `0.2 / 0.99 / 0.9995 / 0.1` |
+| `--stochastic` | 使用随机环境 | 确定性 |
+| `--skip-llm` | 兼容旧用法，等价于 `--mode qlearning` | — |
+
+#### 仅 Q-Learning（实验 7-1，离线，无需 API）
+
+```bash
+python experiment.py --mode qlearning --rl-episodes 10000 --seed 42
+```
+
+训练不到 3 秒即可完成，并打印**学习曲线表格**，直观展现智能体如何在近万局试错中从 0% 胜率逐步学会通关（见下文“实验结果”）。
+
+#### 完整对比（RL vs LLM，实验 7-2）
+
+```bash
+python experiment.py --mode both --model kimi-k3
+```
+
+严格复现正文协议并生成可验收证据，请运行：
+
+```bash
+python run_experiment_7_2.py
+```
+
+该入口固定运行 10,000 局 Q-learning、100 局贪婪评估和且仅一局官方
+Moonshot `kimi-k3` 首次尝试；OpenRouter 替代、API 错误、缺失原始响应
+ID/正文或任何 parser fallback 都会使验收失败。若模型调用已经完成、仅证据
+序列化失败，可运行 `python finalize_experiment_7_2.py <campaign-dir>`，直接从
+已保留的原始结果完成证据，不重复付费调用。
+
+流程：
+
+1. 训练 Q-learning 10000 局（约 3 秒）并打印学习曲线  
+2. 训练 LLM Agent 20 局，并展示详细推理  
+3. 评估双方  
+4. 生成对比图  
+5. 结果写入 `results/`  
+
+**说明**：`experiment.py` 是多局探索入口；正文规范入口只测第一局。2026-07-30
+验收运行包含 17 次串行推理调用，共耗时 416.11 秒，因此没有复现旧版
+“每局 1–2 分钟”的估计。
+
+#### 仅 LLM
+
+```bash
+python experiment.py --mode llm --llm-episodes 20
+```
+
+#### 交互式试玩
+
+```python
+from game_environment import TreasureHuntGame
+
+game = TreasureHuntGame()
+print(game.get_state_description())
+print("Available actions:", game.get_available_actions())
+
+# Try an action
+feedback, reward, done = game.execute_action("take rusty sword")
+print(f"Feedback: {feedback}")
+print(f"Reward: {reward}")
+```
+
+### 验证
+
+在干净环境中运行 pytest 前，先在仓库根目录安装 `dev` extra：
+
+```bash
+uv sync --locked --extra ch1 --extra dev
+
+# 切换目录前先激活环境：
+# macOS/Linux：
+source .venv/bin/activate
+# Windows PowerShell：.venv\Scripts\Activate.ps1
+# Windows cmd：.venv\Scripts\activate.bat
+
+cd chapter1/learning-from-experience
+python -m pytest tests
+```
+
+较长的 Q-learning 学习曲线检查是离线手动 smoke 脚本，不会被默认 pytest 收集：
+
+```bash
+python tests/manual/rl_learning_check.py --episodes 1000
+```
+
+### 实验结果
+
+#### 对比指标
+
+1. **样本效率** — 达到良好表现所需局数、学习速度  
+2. **性能** — 评估胜率、平均回报与回合长度  
+3. **计算成本** — 训练时间；内存（Q 表规模 vs. 经验存储）；LLM 的 API 调用  
+
+#### 可视化
+
+实验会生成对比图，包括：
+
+- 随时间的学习曲线  
+- 胜率演进  
+- 样本效率对比  
+- 关键洞察摘要  
+
+#### 预期结果
+
+##### Q-learning 学习曲线（本地实测，`--mode qlearning --rl-episodes 10000 --seed 42`）
+
+实测学习曲线（确定性环境，胜率按最近 1000 局滑动窗口统计，整段训练约 3 秒）：
+
+| Episodes | Victory rate | Q-table states | epsilon |
+| ---: | ---: | ---: | ---: |
+| 1000 | 0.3% | 123 | 0.606 |
+| 2000 | 0.0% | 123 | 0.368 |
+| 3000 | 0.1% | 126 | 0.223 |
+| 5000 | 0.1% | 128 | 0.100 |
+| 7000 | 97.0% | 138 | 0.100 |
+| 8000 | 99.6% | 138 | 0.100 |
+| 9000 | 99.8% | 139 | 0.100 |
+| 10000 | **98.1%** | 142 | 0.100 |
+
+规范运行训练后 100 局贪婪评估胜率为 **100%**，平均 12 步通关；Kimi K3
+第一局 17 步通关，保留 17/17 条官方响应，零 API 错误、零 fallback，共
+28,242 tokens。它复现了“第一局成功”的实质结论，但没有复现历史记录中的
+Kimi 恰好 18 步和 Q-learning 恰好 11 步。详见[规范证据](validation/20260730_011704/evidence.json)。
+
+##### RL vs LLM（实验 7-2 的对比结论）
+
+- **Q-Learning**：需要近 10000 局才达到稳定通关；把“门/钥匙/剑”当作无意义符号，只能靠统计式暴力探索。  
+- **LLM In-Context**：携带预训练先验，往往第一局就能在十几步内通关；靠推理理解游戏概念结构。  
+- **样本效率**：LLM 高出 2–3 个数量级；但单局推理慢（API 调用 ~1–2 分钟），Q-learning 跑 10000 局只需约 3 秒——权衡取决于交互成本，详见书中实验 7-2。  
+
+### 项目结构
+
+```
+learning-from-experience/
+├── game_environment.py    # Text-based game with hidden mechanics
+├── rl_agent.py            # Q-learning implementation
+├── llm_agent.py           # LLM with in-context learning
+├── experiment.py          # Main experiment runner
+├── demo.py                # Interactive local game demo
+├── quick_demo.py          # Short LLM learning demo
+├── run_experiment_7_2.py  # 正文规范实测与验收门
+├── finalize_experiment_7_2.py # 仅补写证据，不重复 API 调用
+├── env.example            # Optional API-key template
+├── tests/
+│   ├── test_basic.py
+│   ├── test_zero_episodes.py
+│   ├── test_rl_progress_small_episodes.py
+│   └── manual/
+│       └── rl_learning_check.py
+├── requirements.txt       # Python dependencies
+├── README.md              # This file
+└── results/               # Experiment outputs (created on run)
+    └── [timestamp]/
+        ├── rl_agent.pkl           # Trained Q-learning agent
+        ├── llm_experiences.json   # LLM's collected experiences
+        ├── experiment_results.json # Numerical results
+        └── comparison_plots.png   # Visualization
+```
+
+### 技术细节
+
+#### Q-Learning Agent
+
+- **算法**：表格 Q-learning + ε-贪婪探索  
+- **状态表示**：房间、背包与游戏状态的哈希组合  
+- **学习率**：0.2（`--learning-rate`）  
+- **折扣因子**：0.99（`--discount`）  
+- **探索**：ε 从 1.0 起，按 `--epsilon-decay`（0.9995）衰减到 `--epsilon-min`（0.1）  
+
+#### LLM Agent（Kimi K3）
+
+- **模型**：`kimi-k3`（可用 `--model` 或 `MOONSHOT_MODEL` 覆盖）  
+- **推理模型**：Kimi K3 会在最终答案（`message.content`）前输出思维链（`message.reasoning_content`），因此代码使用较大的 `max_tokens=2048`，避免 `ACTION:` 行被思考预算截断。  
+- **学习方式**：上下文学习 + 经验记忆（最多 50 条）  
+- **上下文管理**：存储成功与失败经验  
+- **推理**：行动前提示模型基于过往经验推理  
+- **Temperature**：请求 0.7，但推理模型（Kimi K3、GPT-5）只接受 `temperature=1`，代码会自动强制为 `1`（见 `_reasoning_safe_temperature`）  
+
+### 扩展实验
+
+#### 进一步研究思路
+
+1. **不同游戏**：尝试其他隐藏机制游戏  
+2. **混合方法**：RL 与 LLM 引导结合  
+3. **迁移学习**：测试向相似游戏的迁移  
+4. **消融研究**：去掉推理提示以隔离其影响  
+5. **其他 LLM**：对比不同语言模型  
+
+#### 修改游戏
+
+编辑 `game_environment.py` 可：
+
+- 增加房间与物品  
+- 设计更复杂的隐藏机制  
+- 调整难度与奖励  
+- 加入新类型谜题  
+
+### 教学价值
+
+1. **先验的力量**：语言预训练如何提供有用知识  
+2. **推理 vs. 记忆**：不同学习路径  
+3. **样本效率**：为何对现实任务重要  
+4. **“The Second Half” 论点**：从“能否解决”转向“多高效”  
+
+### 参考文献
+
+- [The Second Half](https://ysymyth.github.io/The-Second-Half/) — Shunyu Yao  
+- [ReAct: Synergizing Reasoning and Acting in Language Models](https://arxiv.org/abs/2210.03629)  
+- Q-learning 原文：Watkins & Dayan (1992)  
+
+---
+
+## Notes / 说明
+
+- Project type: **✅ standalone runnable** (Q-learning offline; LLM needs API key).  
+  项目类型：**✅ 可独立运行**（Q-learning 离线；LLM 需 API Key）。  
+- For educational purposes; inspired by academic work on AI and RL.  
+  教学用途，灵感来自 AI 与强化学习相关研究。  
+- Feel free to add mechanics, other RL algorithms (DQN, PPO, …), providers, or richer metrics.  
+  欢迎增加隐藏机制、其他 RL 算法、提供商或更完善的评估指标。