ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

2026-08-20 13:12:50 +00:00
commit b119135836
10275 changed files with 3284984 additions and 0 deletions
@@ -0,0 +1,32 @@
+# 实验 8-17 GPU 结果（2026-08-07）
+
+## 运行环境
+
+- GPU：NVIDIA RTX PRO 6000 Blackwell Workstation Edition，约 98GB 显存
+- 基座模型：`Qwen/Qwen2.5-7B-Instruct`
+- 训练：LoRA DPO，bf16，4 epochs，学习率 `3e-5`，梯度累积 2，随机种子 717
+- 数据：24 条偏好对；训练数据与未完成任务集、已完成任务保留集的任务完全隔离
+- 训练回执：`train_20260807T163349Z/training_receipt.json`
+
+## 结果
+
+决策边界评估固定“完成”和“继续验证”两个候选动作，比较模型对两个候选的平均 token 对数概率：
+
+| 模型 | 未完成任务集：选对 | 已完成任务保留集：选对 | 未完成任务集平均差值 | 保留集平均差值 |
+|---|---:|---:|---:|---:|
+| 基座 | 3/12（25.0%） | 8/8（100%） | −0.2083 | 4.6904 |
+| LoRA DPO | 11/12（91.7%） | 8/8（100%） | 0.3828 | 2.8525 |
+
+自由生成是补充诊断：未完成任务集的过早结束为 1/12 → 0/12，已完成任务保留集的正常收尾为 6/8 → 0/8。模型在开放式回答里倾向把验证计划写得很长，最终没有明确收尾；这正是为什么主要结论采用与训练提示一致的固定候选比较，而不把自由生成当作唯一指标。
+
+LoRA adapter 约 158MB，未随提交上传；按 README 中的训练命令可以重新生成。训练回执保留实际配置、数据哈希和运行时间；不同版本的 TRL 不一定把最终损失写入同一个日志字段，因此这里不把单个损失值当作实验结论。
+
+## 迭代记录
+
+第一版只看自由生成。基座模型在这组题上本来就很少直接说“完成”，训练前后“过早结束”几乎没有变化，说明这个指标被回答长度和提示遵循掩盖了，不能单独用来判断 DPO 是否学到了收尾决策。
+
+随后把评估改成固定的两个候选动作：“完成：任务已经完成，可以交付”和“继续验证：先运行验收测试并逐条核对验收条件”，比较模型对候选续写的平均 token 对数概率。这个指标直接测量“现在该不该收尾”。训练强度做了小范围对照：`5e-6` 变化不明显；`5e-5` 虽然把候选比较推到 12/12，却让开放式回答明显过度谨慎；最终采用 `3e-5`、4 epochs，在候选比较中得到 11/12，保留集仍为 8/8。还尝试加入 8 条、再减到 4 条“已完成任务控制对”：开放式保留集有所恢复，但未完成任务的候选选择分别掉到 1/12 和 0/12，因此这组控制对没有纳入最终训练集。这个过程保留在各次 `training_receipt.json` 中，避免把一次偶然的参数结果写成结论。
+
+## 解释与限制
+
+这次小规模实验说明，轨迹前缀偏好数据可以把模型在“是否该收尾”上的选择推向正确方向，同时保留已完成任务的正常收尾。它不能证明线上 Coding Agent 的总体成功率会提高：任务只有 24 条，候选比较也不是完整环境回放。实际部署仍应加入更多任务族、真正的隐藏验收测试和通用能力回归。
@@ -0,0 +1,4 @@
+{
+  "run": "train_20260807T163349Z",
+  "training_receipt": "validation/train_20260807T163349Z/training_receipt.json"
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T154228Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "9db5eea4f2434fbb5a4275372c89f8ea144ea175ecec08f2051c500d7685d167",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 5e-06,
+    "epochs": 1,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 16,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T15:42:28.940679+00:00",
+  "finished_at": "2026-08-07T15:42:38.220457+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": null
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T154900Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "1c3fa0ed9a3f8490f5b7817013a81ad2db27c657a5c4f6119dd405584d2cf1fa",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 5e-06,
+    "epochs": 3,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T15:49:00.201304+00:00",
+  "finished_at": "2026-08-07T15:49:18.221031+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": null
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T155308Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "136b3c904fa8cbdb3281481b10f6a6de29dbb29f0669aeb3fc4538791d4a39ba",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 5e-06,
+    "epochs": 3,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T15:53:08.404126+00:00",
+  "finished_at": "2026-08-07T15:53:26.782009+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": null
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T155457Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "136b3c904fa8cbdb3281481b10f6a6de29dbb29f0669aeb3fc4538791d4a39ba",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 5e-05,
+    "epochs": 5,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T15:54:57.705943+00:00",
+  "finished_at": "2026-08-07T15:55:24.918268+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": null
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T155723Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "136b3c904fa8cbdb3281481b10f6a6de29dbb29f0669aeb3fc4538791d4a39ba",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 2e-05,
+    "epochs": 3,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T15:57:23.106371+00:00",
+  "finished_at": "2026-08-07T15:57:41.143566+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": null
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T161544Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "3fe66dc961ad1ed87a66ba8542451132419904211b0c31c074570c3e3ac3df71",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 2e-05,
+    "epochs": 3.0,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T16:15:44.385990+00:00",
+  "finished_at": "2026-08-07T16:16:02.544267+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": 0.0136
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T161751Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "3fe66dc961ad1ed87a66ba8542451132419904211b0c31c074570c3e3ac3df71",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 5e-05,
+    "epochs": 5.0,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T16:17:51.371836+00:00",
+  "finished_at": "2026-08-07T16:18:18.253125+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": 0.0001
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T162013Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "3fe66dc961ad1ed87a66ba8542451132419904211b0c31c074570c3e3ac3df71",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 3e-05,
+    "epochs": 4.0,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T16:20:13.055119+00:00",
+  "finished_at": "2026-08-07T16:20:35.379053+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": 0.0004
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T162703Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "1e9a38963233c79caf5af1f8ac8a1ce8b2f2ae670896516327e2fcebf0a57821",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 3e-05,
+    "epochs": 4.0,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T16:27:03.542276+00:00",
+  "finished_at": "2026-08-07T16:27:26.105120+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": 0.0002
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T162946Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "4559e2bae75a7e71c5c59f7b327a8b59ea1c44ce732ec08d64a50c71a42e03a6",
+  "pair_count": 32,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 3e-05,
+    "epochs": 4.0,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T16:29:46.264964+00:00",
+  "finished_at": "2026-08-07T16:30:14.503330+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": 0.0007
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T163133Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "08aee8de7a25129bef1723236a44fe6aa0f16771c6c35561c935688c63c690e9",
+  "pair_count": 28,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 3e-05,
+    "epochs": 4.0,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T16:31:33.449538+00:00",
+  "finished_at": "2026-08-07T16:31:59.143299+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": 0.0003
+}
@@ -0,0 +1,24 @@
+{
+  "experiment": "8-17 premature-completion-dpo",
+  "run": "train_20260807T163349Z",
+  "model": "Qwen/Qwen2.5-7B-Instruct",
+  "data": "output/preference_pairs.jsonl",
+  "data_sha256": "1e9a38963233c79caf5af1f8ac8a1ce8b2f2ae670896516327e2fcebf0a57821",
+  "pair_count": 24,
+  "config": {
+    "beta": 0.1,
+    "learning_rate": 3e-05,
+    "epochs": 4.0,
+    "lora_r": 16,
+    "lora_alpha": 32,
+    "per_device_batch_size": 1,
+    "gradient_accumulation": 2,
+    "bf16": true,
+    "gradient_checkpointing": true,
+    "seed": 717
+  },
+  "started_at": "2026-08-07T16:33:49.602096+00:00",
+  "finished_at": "2026-08-07T16:34:12.355041+00:00",
+  "adapter_dir": "output/adapter",
+  "training_loss": 0.0002
+}