ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
@@ -0,0 +1,9 @@
|
||||
node_modules/
|
||||
output/
|
||||
slidev_workspace/exports/
|
||||
slidev_workspace/slides.md
|
||||
slidev_workspace/.slidev/
|
||||
slidev_workspace/public/*.png
|
||||
__pycache__/
|
||||
.env
|
||||
package-lock.json
|
||||
@@ -0,0 +1,369 @@
|
||||
# Experiment 5-4: Paper → PPT (Proposer–Reviewer) / 实验 5-4：基于论文的 PPT 自动生成（提议者-审核者机制）
|
||||
|
||||
> Companion lab for *AI Agents in Depth*, Chapter 5 — generate Slidev decks from a paper; Proposer writes code, Reviewer renders PNG and reviews with Vision LLM.
|
||||
> 《深入理解 AI Agent》第 5 章：把「做 PPT」重构为代码生成；Proposer 写 Slidev，Reviewer 真渲染 PNG 并用 Vision 审查迭代。
|
||||
|
||||
← [Chapter 5 index / 返回第 5 章目录](../README.md)
|
||||
|
||||
---
|
||||
|
||||
## English
|
||||
|
||||
### One-line takeaway
|
||||
|
||||
Proposer only writes Slidev code; Reviewer **renders each page to PNG** and uses a **Vision LLM** to flag issues (text overflow / overcrowding / image size). Proposer revises from structured feedback. Versus single-agent self-review (stacking every rendered image in one context), dual-agent **peak context is much smaller**—Proposer never sees images; Reviewer each round only sees the latest screenshots.
|
||||
|
||||
The canonical completed run is
|
||||
[`validation/runs/exp5-4-real-pdf-both-20260730-v9/comparison_summary.json`](validation/runs/exp5-4-real-pdf-both-20260730-v9/comparison_summary.json)
|
||||
(SHA-256 `bfd913d311ab4d6ad5a8cae93b61ce54ce6d19f9d2d10ee2afdef06becd1e09f`).
|
||||
Both twenty-page decks used the pinned real PDF and three original,
|
||||
provenance-tracked figure crops, rendered every page, and scored 95/pass under
|
||||
the same independent Vision judge. Quality tied; dual-agent peak context was
|
||||
24,186 tokens versus 92,601 for single-agent self-review (3.83×), with total
|
||||
usage 73,227 versus 298,259 tokens. Every formal gate is true.
|
||||
|
||||
### Why render before judging
|
||||
|
||||
When the Agent finishes Slidev source it **does not know the real layout**: crowding, overflow, image size only appear after pixel render. Reviewer therefore receives **new information** the Proposer never saw—the value of the mechanism.
|
||||
|
||||
### Proposer–Reviewer split
|
||||
|
||||
| Role | Duty | Context contents |
|
||||
|---|---|---|
|
||||
| **Proposer** (configured text model) | Read paper → plan pages → write/revise `slides.md` | Paper text + **accumulated structured text feedback** (never images) |
|
||||
| **Reviewer** (configured Vision model) | Look at latest per-page PNGs; structured JSON advice | **Fresh call each round**, latest screenshots only |
|
||||
|
||||
Reviewer advice is structured and actionable, not vague “looks bad”: fields `page`, `issue_type` (`text_overflow` / `overcrowded` / `image_size` / `readability` / `layout`), `severity` (high/medium/low), `suggestion`, plus deck-level `overall_score` and `pass`.
|
||||
|
||||
Loop: feedback → revise → re-submit until `pass` or max rounds.
|
||||
|
||||
### Ablation: single-agent self-review vs dual-agent
|
||||
|
||||
`demo.py` runs both and scores both final decks with the **same independent Vision judge** (comparable quality):
|
||||
|
||||
- **A dual-agent**: as above. Proposer context grows in text only; Reviewer resets each round.
|
||||
- **B single-agent self-review**: one agent in **one conversation** generates → sees its own renders → revises. Past images **stay in context** and inflate quickly (book: “context blows past limits”).
|
||||
|
||||
The script prints per-call prompt token series, totals, and **peak context** (max single prompt tokens). More pages/rounds → larger B vs A peak gap.
|
||||
|
||||
### Run
|
||||
|
||||
```bash
|
||||
# 1) From the repository root: Python deps
|
||||
uv sync --locked --python 3.12 --extra ch5
|
||||
|
||||
# Activate it before changing directories:
|
||||
# macOS/Linux:
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell: .\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd: .venv\Scripts\activate.bat
|
||||
|
||||
# pip fallback when uv is not installed:
|
||||
# python -m pip install -e ".[ch5]"
|
||||
|
||||
cd chapter5/paper-to-ppt
|
||||
|
||||
# Single-project compatibility path, still supported during migration:
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
# 2) Slidev + render deps (Node). First time ~1–2 min:
|
||||
npm install
|
||||
# - @slidev/cli
|
||||
# - playwright-chromium (under slidev export --format png)
|
||||
# - typescript (twoslash highlighting; else export may ERR_MODULE_NOT_FOUND)
|
||||
# If chromium binary missing:
|
||||
# npx playwright install chromium
|
||||
|
||||
# 3) Keys
|
||||
cp env.example .env # OPENAI_API_KEY (or OPENROUTER_API_KEY fallback)
|
||||
|
||||
# 4) Canonical manuscript campaign: pinned real arXiv PDF, three original
|
||||
# PDF figures, both comparison arms, real Slidev rendering and Vision review
|
||||
python demo.py --provider ark --text-model doubao-seed-1-6-250615 \
|
||||
--vision-model doubao-seed-1-6-250615 --mode both --max-rounds 4 \
|
||||
--out-dir validation/runs/my-real-run
|
||||
```
|
||||
|
||||
#### Common flags (`python demo.py --help`)
|
||||
|
||||
A full run may call gpt-5.6-luna Vision dozens of times (slow/costly). Flags for faster paths, other papers, output dirs, models:
|
||||
|
||||
| Flag | Role |
|
||||
|---|---|
|
||||
| `--paper PATH` | Legacy/non-canonical local Markdown input. Omit it for the pinned real arXiv PDF used by the formal campaign. |
|
||||
| `--out-dir DIR` | Artifacts dir (default `output/`): per-round `slides.md` / `review.json` / `comparison_summary.json`. Rendered PNGs always under `slidev_workspace/exports/` |
|
||||
| `--text-model NAME` | Proposer / single-agent text model; overrides `TEXT_MODEL` (default `gpt-5.6-luna`) |
|
||||
| `--vision-model NAME` | Reviewer / judge vision model (must support images); overrides `VISION_MODEL` (default `gpt-5.6-luna`) |
|
||||
| `--mode {both,dual,single}` | One scheme only (`dual` / `single`) to cut time/cost; `both` (default) for cross-scheme compare |
|
||||
| `--max-rounds N` | Max iterations per scheme (default 3). `--max-rounds 1` = first draft only—fastest real-LLM smoke |
|
||||
| `--dry-run` | **Offline** Proposer–Reviewer loop: real render of two scripted `slides.md` (crowded draft → split revision); **deterministic heuristics** (char count per page, not Vision LLM) as Reviewer. **No LLM, no API key** |
|
||||
| `--smoke` | **Only** Slidev render path (2-page deck); **no LLM, no API key** |
|
||||
|
||||
```bash
|
||||
python demo.py --smoke # free: Node/Slidev/chromium OK?
|
||||
python demo.py --dry-run # free: offline dual-agent loop with real renders
|
||||
python demo.py --mode dual --max-rounds 1 # one real LLM smoke (needs API key)
|
||||
python demo.py --paper my_paper.md --out-dir run_my
|
||||
```
|
||||
|
||||
> In `--dry-run`, both `slides.md` versions are **scripted** (not LLM); Reviewer is a **heuristic** on char counts—not Vision. It only runs the **structure** of the loop offline and produces real PNGs. For real pixel review with gpt-5.6-luna use `python demo.py` (needs `OPENAI_API_KEY`). One offline dry-run: draft 4 pages (pages 2/3/4 high overcrowded, score=55, pass=False) → revised 18 pages (score=100, pass=True); PNGs under `slidev_workspace/exports/dryrun_round*/`.
|
||||
|
||||
### Files
|
||||
|
||||
| File | Role |
|
||||
|---|---|
|
||||
| `demo.py` | Main: both schemes, independent judge, token comparison |
|
||||
| `agents.py` | `Proposer` / `Reviewer` / `SelfReviewAgent` + `TokenMeter` |
|
||||
| `renderer.py` | `slidev export --format png` → per-page PNGs |
|
||||
| `paper_source.py` | Downloads the hash-pinned paper PDF, extracts its text, and crops three original paper figures with provenance |
|
||||
| `make_figures.py` / `paper/sample_paper.md` | Legacy local-Markdown compatibility path; never satisfies the formal campaign gate |
|
||||
| `package.json` | Slidev + render deps |
|
||||
| `output/` | Per-round `slides.md`, `review.json`, `comparison_summary.json` |
|
||||
| `slidev_workspace/exports/` | Per-round PNG folders (`dual_round1/`, `single_round1/`, …) |
|
||||
|
||||
### Sample outputs
|
||||
|
||||
After a full run (excerpt of real artifacts):
|
||||
|
||||
```
|
||||
output/
|
||||
├── dual_round1_slides.md
|
||||
├── dual_round1_review.json
|
||||
├── dual_round2_slides.md
|
||||
├── dual_round2_review.json
|
||||
├── dual_round3_slides.md
|
||||
├── single_round1_slides.md
|
||||
├── single_round2_slides.md
|
||||
├── single_round3_slides.md
|
||||
└── comparison_summary.json
|
||||
|
||||
slidev_workspace/exports/
|
||||
├── dual_round1/1.png … 5.png
|
||||
├── dual_round2/1.png … 8.png
|
||||
└── single_round1/1.png …
|
||||
```
|
||||
|
||||
> Slidev PNG export is **one PNG per page** (`1.png`, `2.png`, …), not a single PDF; change `--format png` to `--format pdf` in `renderer.py` if needed. `comparison_summary.json` holds `iteration_scores`, `final_quality`, and `peak_context_prompt_tokens`—the book’s core comparison numbers.
|
||||
|
||||
### Adapt / extend
|
||||
|
||||
- **Model / provider**: env (`env.example`) or CLI (CLI wins); no code change.
|
||||
- `OPENAI_API_KEY` (or `OPENROUTER_API_KEY` fallback).
|
||||
- `OPENAI_BASE_URL`: any OpenAI-compatible endpoint.
|
||||
- `TEXT_MODEL` / `--text-model` (default `gpt-5.6-luna`).
|
||||
- `VISION_MODEL` / `--vision-model` must support images (default `gpt-5.6-luna`).
|
||||
- **Paper / out dir**: omit `--paper` for the canonical hash-pinned arXiv PDF; `--paper PATH` is a legacy compatibility path that cannot pass the formal source-provenance gate. `--out-dir DIR` selects the evidence directory.
|
||||
- **Slidev deps**: need **Node** + local `node_modules/` (`@slidev/cli`, `playwright-chromium`, `typescript`). Re-run `npm install`; if browser binary missing, `npx playwright install chromium`. Then `python demo.py --smoke` before a full run.
|
||||
|
||||
### Canonical source and layout preflight
|
||||
|
||||
The formal mode does not manufacture an intentionally bad first draft.
|
||||
Before paying for pixel review, a deterministic source preflight requires
|
||||
18–20 pages, at most four bullets per page, dedicated source-figure pages,
|
||||
short one-line figure titles, and bounded inline image layout. Vision remains
|
||||
the authority for actual overflow, readability and layout. The two attention
|
||||
figures retain the published pixels and exact PDF crop rectangles; a recorded
|
||||
90-degree presentation transform makes the original vertical labels readable
|
||||
on a landscape slide. Failed refinement runs remain under `validation/runs/`
|
||||
and are never promoted when the independent judge reports a blocking defect.
|
||||
|
||||
### Limitations
|
||||
|
||||
- **Subjective taste**: Reviewer preferences ≠ user preferences; may converge to Reviewer-local optima (see book thinking questions).
|
||||
- **Input modes**: the canonical mode parses a real hash-pinned PDF and uses three crops from that PDF. `--paper PATH` deliberately remains a non-canonical compatibility mode with programmatic figures.
|
||||
- **Cost/time**: up to 20 screenshots per Reviewer round; screenshots are scaled to 1280px wide before the Vision call.
|
||||
- **Non-determinism**: LLM/Vision scores vary; `temperature` lowered but “pass in 1 round” depends on first draft.
|
||||
- **Render deps**: `slidev export` needs playwright-chromium; fix binary issues before running (see step 2).
|
||||
|
||||
---
|
||||
|
||||
## 中文
|
||||
|
||||
### 一句话结论
|
||||
|
||||
Proposer 只写 Slidev 代码、Reviewer 真正把每页**渲染成 PNG** 再用 **Vision LLM 看图**
|
||||
挑毛病（文字溢出 / 内容拥挤 / 图片尺寸），Proposer 据结构化反馈迭代修订。相比"单 Agent
|
||||
自审"（把历次渲染图片都堆在同一上下文里），双 Agent 分工的**上下文峰值显著更小**——
|
||||
因为 Proposer 全程不看图片、Reviewer 每轮只看最新一版截图。
|
||||
|
||||
### 为什么需要"渲染出来再看"
|
||||
|
||||
Agent 写完 Slidev 代码时**并不知道实际渲染效果**：内容会不会太挤、文字会不会溢出、
|
||||
图片尺寸是否合适——这些只有真正渲染成像素才看得出来。所以 Reviewer 接触到的是
|
||||
Proposer 看不到的**新信息**（渲染结果），这正是本机制的价值所在。
|
||||
|
||||
### 提议者-审核者分工
|
||||
|
||||
| 角色 | 职责 | 上下文里有什么 |
|
||||
|---|---|---|
|
||||
| **Proposer**（`gpt-5.6-luna`，纯文本） | 读论文 → 规划页面 → 生成/修订 `slides.md` | 论文正文 + **累积的结构化文字反馈**（永不含图片） |
|
||||
| **Reviewer**（`gpt-5.6-luna`，Vision） | 看最新一版每页 PNG，输出结构化建议 JSON | **每轮全新调用**，只含最新一版截图 |
|
||||
|
||||
Reviewer 的建议是结构化、可执行的，而非模糊的"不好看"，包含字段：
|
||||
`page`（页码）、`issue_type`（`text_overflow`/`overcrowded`/`image_size`/`readability`/`layout`）、
|
||||
`severity`（high/medium/low）、`suggestion`（具体修改建议）、以及整份的 `overall_score` 与 `pass`。
|
||||
|
||||
Proposer 收到反馈 → 理解意图 → 修订代码 → 再次提交 Reviewer，循环直到 `pass` 或达最大轮数。
|
||||
|
||||
### 对照实验：单 Agent 自审 vs 双 Agent 分工
|
||||
|
||||
`demo.py` 同时跑两种方案，并用**同一位独立 Vision 评委**给两者的最终 PPT 打分（保证质量可比）：
|
||||
|
||||
- **方案 A 双 Agent**：如上。Proposer 上下文只增文本；Reviewer 每轮重置、只看最新截图。
|
||||
- **方案 B 单 Agent 自审**：一个 Agent 在**同一段对话**里生成 → 看自己的渲染截图自审 → 修订。
|
||||
历次渲染的图片会**一直留在上下文里**，随迭代快速膨胀（书中所述"上下文迅速超限"）。
|
||||
|
||||
脚本打印每次调用的 prompt token 序列、总量、以及**上下文峰值**（单次 prompt token，
|
||||
决定是否撑爆上下文窗口）。页数越多、迭代越多，方案 B 的峰值相对方案 A 越夸张。
|
||||
|
||||
### 运行
|
||||
|
||||
```bash
|
||||
# 1) 在仓库根目录安装 Python 依赖
|
||||
uv sync --locked --python 3.12 --extra ch5
|
||||
|
||||
# 切换目录前先激活环境：
|
||||
# macOS/Linux：
|
||||
source .venv/bin/activate
|
||||
# Windows PowerShell：.\.venv\Scripts\Activate.ps1
|
||||
# Windows cmd：.venv\Scripts\activate.bat
|
||||
|
||||
# 未安装 uv 时可用 pip 兜底：
|
||||
# python -m pip install -e ".[ch5]"
|
||||
|
||||
cd chapter5/paper-to-ppt
|
||||
|
||||
# 迁移期间仍支持单项目兼容路径：
|
||||
# python -m pip install -r requirements.txt
|
||||
|
||||
# 2) Slidev + 渲染依赖（Node）。首次约 1-2 分钟：
|
||||
npm install
|
||||
# - @slidev/cli：Slidev 命令行
|
||||
# - playwright-chromium：slidev export --format png 的底层浏览器
|
||||
# - typescript：Slidev 的 twoslash 代码高亮所需（否则 export 会 ERR_MODULE_NOT_FOUND）
|
||||
# 若 npm install 没有自动装好 chromium 浏览器二进制，运行：
|
||||
# npx playwright install chromium
|
||||
|
||||
# 3) 配置 Key
|
||||
cp env.example .env # 填入 OPENAI_API_KEY（未配置时设 OPENROUTER_API_KEY 自动改走 OpenRouter）
|
||||
|
||||
# 4) 正式活动：固定哈希的真实 arXiv PDF、三张原论文图、两种对照方案、
|
||||
# 真实 Slidev 渲染与 Vision 审查
|
||||
python demo.py --provider ark --text-model doubao-seed-1-6-250615 \
|
||||
--vision-model doubao-seed-1-6-250615 --mode both --max-rounds 4 \
|
||||
--out-dir validation/runs/my-real-run
|
||||
```
|
||||
|
||||
#### 常用参数（`python demo.py --help`）
|
||||
|
||||
一次完整运行会做数十次 gpt-5.6-luna Vision 调用，较慢较贵。下列参数提供更快的路径，并允许更换论文、输出目录与模型：
|
||||
|
||||
| 参数 | 作用 |
|
||||
|---|---|
|
||||
| `--paper PATH` | 非正式兼容入口：使用本地 Markdown。正式活动须省略该参数，以使用固定哈希的真实 arXiv PDF。 |
|
||||
| `--out-dir DIR` | 产物输出目录（默认 `output/`）：各轮 `slides.md`/`review.json`/`comparison_summary.json`。渲染 PNG 始终在 `slidev_workspace/exports/`。 |
|
||||
| `--text-model NAME` | Proposer / 单 Agent 文本模型，**覆盖** `TEXT_MODEL` 环境变量（默认 `gpt-5.6-luna`）。 |
|
||||
| `--vision-model NAME` | Reviewer / 独立评委看图模型（须支持图像），**覆盖** `VISION_MODEL` 环境变量（默认 `gpt-5.6-luna`）。 |
|
||||
| `--mode {both,dual,single}` | 只跑一种方案（`dual`=提议者-审核者，`single`=单 Agent 自审），省一半时间/费用；`both`（默认）才做跨方案对比。 |
|
||||
| `--max-rounds N` | 每种方案的最大迭代轮数（默认 3）。`--max-rounds 1` 只出首版、不修订，是最快的**真实 LLM** 冒烟。 |
|
||||
| `--dry-run` | **离线走通提议者-审核者循环**：真实渲染两版脚本化 `slides.md`（拥挤初稿→拆页修订稿），用**确定性启发式规则**（按每页文字量判定，非 Vision LLM）扮演 Reviewer，完整展示“生成→渲染→审查→修订”闭环。**不调用任何 LLM、无需 API Key**。 |
|
||||
| `--smoke` | **只**验证 Slidev 渲染链路（渲染一个两页 deck），**不调用任何 LLM、无需 API Key**。最快的“没搞坏渲染”自检。 |
|
||||
|
||||
```bash
|
||||
python demo.py --smoke # 不花钱，验证 Node/Slidev/chromium 可用
|
||||
python demo.py --dry-run # 不花钱，离线看清提议者-审核者闭环（真实渲染）
|
||||
python demo.py --mode dual --max-rounds 1 # 一次真实 LLM 冒烟（需 API Key）
|
||||
python demo.py --paper my_paper.md --out-dir run_my # 换论文、换输出目录
|
||||
```
|
||||
|
||||
> `--dry-run` 里的两版 `slides.md` 是**脚本化**的（不是 LLM 生成），Reviewer 也只是按字符数判定拥挤的**启发式规则**、并非 Vision LLM——它只用来在没有 API Key 时把闭环**结构**跑通、产出真实渲染的 PNG。要看 gpt-5.6-luna **真的看像素**审查，请用 `python demo.py`（需 `OPENAI_API_KEY`）。一次离线 dry-run 的真实结果：初稿 4 页（第 2/3/4 页被判 high 级 overcrowded、score=55、pass=False）→ 拆页修订稿 18 页（score=100、pass=True），渲染 PNG 见 `slidev_workspace/exports/dryrun_round*/`。
|
||||
|
||||
### 文件说明
|
||||
|
||||
| 文件 | 作用 |
|
||||
|---|---|
|
||||
| `demo.py` | 主流程：跑两种方案、独立评委打分、打印 token 对比 |
|
||||
| `agents.py` | `Proposer` / `Reviewer` / `SelfReviewAgent` 三个 Agent + `TokenMeter` 计量 |
|
||||
| `renderer.py` | 调 `slidev export --format png` 把 `slides.md` 渲染成逐页 PNG |
|
||||
| `paper_source.py` | 下载固定哈希的真实论文 PDF、直接提取正文，并裁出三张带来源信息的原论文图 |
|
||||
| `paper/sample_paper.md` / `make_figures.py` | 旧版本地 Markdown 兼容路径；不会通过正式实验的来源门禁 |
|
||||
| `package.json` | Slidev 与渲染依赖 |
|
||||
| `output/` | 运行产物：各轮 `slides.md`、`review.json`、`comparison_summary.json` |
|
||||
| `slidev_workspace/exports/` | 各轮渲染出的 PNG（`dual_round1/`、`single_round1/` …） |
|
||||
|
||||
### 预期输出示例
|
||||
|
||||
一次完整运行后，`output/` 与 `slidev_workspace/exports/` 下的真实产物（节选）：
|
||||
|
||||
```
|
||||
output/
|
||||
├── dual_round1_slides.md # 双 Agent 第 1 版 slidev 源码（首版故意很挤）
|
||||
├── dual_round1_review.json # Reviewer 对第 1 版的结构化建议 JSON
|
||||
├── dual_round2_slides.md # 据反馈修订后的第 2 版
|
||||
├── dual_round2_review.json
|
||||
├── dual_round3_slides.md
|
||||
├── single_round1_slides.md # 单 Agent 自审各版
|
||||
├── single_round2_slides.md
|
||||
├── single_round3_slides.md
|
||||
└── comparison_summary.json # 两方案质量分 + token 消耗汇总
|
||||
|
||||
slidev_workspace/exports/
|
||||
├── dual_round1/1.png … 5.png # 首版渲染：段落太长、图表底部超出页面
|
||||
├── dual_round2/1.png … 8.png # 修订版：拆页后每页 8 张更干净
|
||||
└── single_round1/1.png … # 单 Agent 各版渲染
|
||||
```
|
||||
|
||||
> 说明：Slidev 的 PNG 导出是**逐页一张 PNG**（`1.png`、`2.png`…），本实验不产出单一 PDF；
|
||||
> 如需 PDF，可把 `renderer.py` 里的 `--format png` 改为 `--format pdf`。
|
||||
> `comparison_summary.json` 里记录两方案的 `iteration_scores`、`final_quality` 与
|
||||
> `peak_context_prompt_tokens`（上下文峰值），即书中的核心对比数据。
|
||||
|
||||
### 如何适配 / 扩展
|
||||
|
||||
- **换模型 / 换供应商**：通过环境变量（见 `env.example`）或命令行参数（优先级更高），代码无需改动。
|
||||
- `OPENAI_API_KEY`：密钥（必填其一；未配置时用 `OPENROUTER_API_KEY` 兜底，自动改走 OpenRouter）。
|
||||
- `OPENAI_BASE_URL`：指向任何兼容 OpenAI 协议的端点（自建网关 / 其它供应商）。
|
||||
- `TEXT_MODEL` / `--text-model`：Proposer / 单 Agent 文本部分用的模型（默认 `gpt-5.6-luna`）。
|
||||
- `VISION_MODEL` / `--vision-model`：Reviewer / 独立评委看图用的模型，**必须支持图像输入**（默认 `gpt-5.6-luna`）。
|
||||
- **换输入论文 / 输出目录**：正式活动省略 `--paper`，使用固定哈希的真实 arXiv PDF 与三张原论文图；
|
||||
`--paper my.md` 只用于兼容自定义 Markdown，不会被标记为正式完成。`--out-dir DIR` 指定证据目录。
|
||||
- **Slidev 渲染依赖（重要）**：渲染链路依赖 **Node** + 本目录内 `node_modules/`，其中包含
|
||||
`@slidev/cli`、`playwright-chromium`（`slidev export --format png` 的底层浏览器）、`typescript`
|
||||
（twoslash 代码高亮所需）。若 `node_modules/` 缺失或损坏，在本目录执行 `npm install` 重装；
|
||||
若浏览器二进制没装好，补跑 `npx playwright install chromium`。装好后先 `python demo.py --smoke`
|
||||
验证渲染链路，再跑完整流程。
|
||||
|
||||
### 关于"第一版故意写得很挤"
|
||||
|
||||
为了**稳定复现**"渲染 → 发现问题 → 修订"的闭环，`agents.py` 里让 Proposer/单 Agent 的
|
||||
**首版**先把整篇论文塞进约 4 页、成段贴原文（一种常见的"先把内容倒进去"的初稿写法）。
|
||||
这会产生**真实的**文字溢出与图表被裁切（见 `slidev_workspace/exports/dual_round1/2.png`：
|
||||
段落太长、图表底部超出页面）。Reviewer 的问题都是视觉模型（默认 gpt-5.6-luna）**看真实像素**得出的，修订也是
|
||||
真实的——不是预设脚本。若把首版指令改成"直接生成 8-12 页精简版"，视觉模型往往一版就过关，
|
||||
反而看不到迭代过程。一次真实运行的结果（会有随机波动）：
|
||||
|
||||
```
|
||||
双 Agent：round1 score=85 pass=False（4 个 medium：p2/p3/p4 overcrowded、p2 image_size）
|
||||
→ Proposer 拆页精简 → round2 score=95 pass=True（+10 改善）
|
||||
上下文峰值：双 Agent = 9308 tok，单 Agent 自审 = 14179 tok（单 Agent 图片累积：1640→8069→14179）
|
||||
```
|
||||
|
||||
### 局限
|
||||
|
||||
- **审美主观**：Reviewer 的偏好未必等于目标用户的偏好，反馈循环可能收敛到 Reviewer
|
||||
认可但用户嫌挤的局部最优（见书末思考题：如何让用户偏好也进入循环）。
|
||||
- **输入模式**：正式模式解析固定哈希的真实 PDF，并直接裁出三张原论文图；只有显式使用
|
||||
`--paper PATH` 时才走程序化图表的旧版兼容路径。
|
||||
- **成本/时长**：每轮 Reviewer 要把约 10 张截图发给视觉模型（默认 gpt-5.6-luna），单次运行需数十次
|
||||
API 调用；已把截图统一缩放到 1280px 宽以控制 token。
|
||||
- **确定性**：LLM 与 Vision 判定有随机性，具体分数/建议每次略有不同；`temperature`
|
||||
已调低，但迭代是否恰好"1 轮达标"取决于首版质量。
|
||||
- **渲染依赖**：`slidev export` 依赖 playwright-chromium；无网络/无法装 chromium 的
|
||||
环境需先解决浏览器二进制问题（见"运行"第 2 步）。
|
||||
|
||||
---
|
||||
|
||||
## Notes / 说明
|
||||
|
||||
- Start with `--smoke` / `--dry-run` before a full Vision run. / 完整 Vision 跑前先 `--smoke` / `--dry-run`。
|
||||
- Commands/code/paths/env vars are identical in both language sections. / 命令、代码、路径与环境变量在中英文两侧保持一致。
|
||||
@@ -0,0 +1,507 @@
|
||||
"""
|
||||
提议者（Proposer）与审核者（Reviewer）两个 Agent，以及一个带 token 计量的 LLM 客户端。
|
||||
|
||||
设计要点（对应书中“提议者-审核者”机制）：
|
||||
- Proposer 只处理**文本**：论文正文 + 累积的结构化文字反馈；从不接收渲染图片。
|
||||
- Reviewer 每一轮**只看最新一版的渲染截图**，且每轮都是一次全新的、无历史的调用。
|
||||
- 单 Agent 自审对照组则相反：同一段对话里不断累积历次渲染的图片，上下文迅速膨胀。
|
||||
|
||||
所有对 OpenAI 的调用都经过 TokenMeter 统计 prompt / completion token，
|
||||
用于最后的“单 Agent vs 双 Agent 上下文消耗”对比。
|
||||
"""
|
||||
import base64
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from openai import OpenAI
|
||||
from PIL import Image
|
||||
|
||||
# 文本生成用的模型（Proposer / 单 Agent 的文本部分）
|
||||
TEXT_MODEL = os.environ.get("TEXT_MODEL", "gpt-5.6-luna")
|
||||
# 视觉审查用的模型（Reviewer / 单 Agent 的看图部分），必须支持图像
|
||||
VISION_MODEL = os.environ.get("VISION_MODEL", "gpt-5.6-luna")
|
||||
|
||||
OPENROUTER_BASE_URL = "https://openrouter.ai/api/v1"
|
||||
PROVIDER = os.environ.get("PPT_PROVIDER", "auto")
|
||||
|
||||
|
||||
def configure_provider(provider: str) -> None:
|
||||
global PROVIDER
|
||||
PROVIDER = provider
|
||||
|
||||
|
||||
def map_model_to_openrouter(model: str) -> str:
|
||||
"""把直连模型名映射为 OpenRouter 上的 id（非可映射 id 统一兜底到当前廉价旗舰）。"""
|
||||
if not model or "/" in model:
|
||||
return model or "openai/gpt-5.6-luna"
|
||||
m = model.lower()
|
||||
if m.startswith(("gpt-", "o1", "o3", "o4")):
|
||||
return "openai/" + model
|
||||
if m.startswith("claude"):
|
||||
if "haiku" in m:
|
||||
return "anthropic/claude-haiku-4.5"
|
||||
if "sonnet" in m:
|
||||
return "anthropic/claude-sonnet-4.6"
|
||||
return "anthropic/claude-opus-4.8"
|
||||
if m.startswith("gemini"):
|
||||
return "google/" + model
|
||||
return "openai/gpt-5.6-luna"
|
||||
|
||||
# 发送给 Vision 前把截图缩放到该宽度，兼顾“看得清文字溢出”与“控制 token 成本”
|
||||
VISION_IMAGE_WIDTH = 1280
|
||||
|
||||
FIGURE_PAGE_TITLES = {
|
||||
"paper_figure_1_transformer.png": "Transformer Architecture (Figure 1)",
|
||||
"paper_figure_3_long_distance.png": "Long-Distance Attention (Figure 3)",
|
||||
"paper_figure_4_anaphora.png": "Anaphora Attention (Figure 4)",
|
||||
}
|
||||
|
||||
|
||||
class TokenMeter:
|
||||
"""累计一个“角色/模式”消耗的 token，并记录每次调用的 prompt token（用于看上下文峰值）。"""
|
||||
|
||||
def __init__(self, name: str):
|
||||
self.name = name
|
||||
self.prompt_tokens = 0
|
||||
self.completion_tokens = 0
|
||||
self.calls = 0
|
||||
self.peak_prompt_tokens = 0 # 单次调用最大的 prompt token —— 决定是否“撑爆上下文”
|
||||
self.per_call_prompt = []
|
||||
self.receipts = []
|
||||
|
||||
def add(self, response, request: dict, latency_s: float | None = None):
|
||||
usage = response.usage
|
||||
self.calls += 1
|
||||
pt = usage.prompt_tokens
|
||||
self.prompt_tokens += pt
|
||||
self.completion_tokens += usage.completion_tokens
|
||||
self.peak_prompt_tokens = max(self.peak_prompt_tokens, pt)
|
||||
self.per_call_prompt.append(pt)
|
||||
choice = response.choices[0]
|
||||
receipt = {
|
||||
"meter": self.name,
|
||||
"called_at_utc": dt.datetime.now(dt.timezone.utc).isoformat(),
|
||||
"latency_s": round(latency_s, 3) if latency_s is not None else None,
|
||||
"request": _sanitize_for_evidence(request),
|
||||
"response": {
|
||||
"id": response.id,
|
||||
"model": response.model,
|
||||
"finish_reason": choice.finish_reason,
|
||||
"content": choice.message.content,
|
||||
},
|
||||
"usage": {
|
||||
"prompt_tokens": usage.prompt_tokens,
|
||||
"completion_tokens": usage.completion_tokens,
|
||||
"total_tokens": usage.total_tokens,
|
||||
"cached_prompt_tokens": getattr(
|
||||
getattr(usage, "prompt_tokens_details", None), "cached_tokens", None
|
||||
),
|
||||
},
|
||||
}
|
||||
self.receipts.append(receipt)
|
||||
checkpoint = os.environ.get("PPT_RECEIPT_CHECKPOINT")
|
||||
if checkpoint:
|
||||
path = Path(checkpoint)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
existing = []
|
||||
if path.is_file():
|
||||
existing = json.loads(path.read_text(encoding="utf-8"))
|
||||
existing.append(receipt)
|
||||
temporary = path.with_suffix(path.suffix + ".tmp")
|
||||
temporary.write_text(
|
||||
json.dumps(existing, ensure_ascii=False, indent=2), encoding="utf-8"
|
||||
)
|
||||
temporary.replace(path)
|
||||
|
||||
@property
|
||||
def total_tokens(self):
|
||||
return self.prompt_tokens + self.completion_tokens
|
||||
|
||||
|
||||
def _client() -> OpenAI:
|
||||
# 通用 OpenRouter 兜底：无直连 key，或默认 gpt-5.x（直连需组织实名认证）时改走 OpenRouter。
|
||||
global TEXT_MODEL, VISION_MODEL
|
||||
if PROVIDER == "ark":
|
||||
api_key = os.environ.get("ARK_API_KEY")
|
||||
base_url = "https://ark.cn-beijing.volces.com/api/v3"
|
||||
elif PROVIDER == "moonshot":
|
||||
api_key = os.environ.get("MOONSHOT_API_KEY")
|
||||
base_url = "https://api.moonshot.cn/v1"
|
||||
elif PROVIDER == "openrouter":
|
||||
api_key = os.environ.get("OPENROUTER_API_KEY")
|
||||
base_url = OPENROUTER_BASE_URL
|
||||
else:
|
||||
api_key = os.environ.get("OPENAI_API_KEY")
|
||||
base_url = os.environ.get("OPENAI_BASE_URL")
|
||||
orkey = os.environ.get("OPENROUTER_API_KEY")
|
||||
prefer_or = PROVIDER == "auto" and bool(orkey) and (
|
||||
(TEXT_MODEL or "").lower().startswith("gpt-5") or (VISION_MODEL or "").lower().startswith("gpt-5")
|
||||
)
|
||||
if PROVIDER == "openrouter" or prefer_or or (PROVIDER == "auto" and not api_key and orkey):
|
||||
api_key, base_url = orkey, OPENROUTER_BASE_URL
|
||||
# 走 OpenRouter 时把模型名映射为其 id（幂等：已带前缀的 id 原样返回）。
|
||||
TEXT_MODEL = map_model_to_openrouter(TEXT_MODEL)
|
||||
VISION_MODEL = map_model_to_openrouter(VISION_MODEL)
|
||||
if not api_key:
|
||||
raise SystemExit(
|
||||
"❌ 未检测到 OPENAI_API_KEY（或 OPENROUTER_API_KEY 兜底）。请先 `cp env.example .env` 并填入有效的 "
|
||||
"OpenAI API Key（或 `export OPENAI_API_KEY=your-openai-api-key` / `export OPENROUTER_API_KEY=...`）后再运行。"
|
||||
)
|
||||
# timeout + max_retries：单次网络抖动/SSL 中断会自动重试，而不是让整条流水线崩溃。
|
||||
return OpenAI(
|
||||
api_key=api_key,
|
||||
base_url=base_url,
|
||||
# Large single-agent requests carry the extracted paper plus several
|
||||
# full slide revisions (and later image history). Ark can legitimately
|
||||
# take more than one minute to produce the complete Markdown deck.
|
||||
# Keep bounded retries, but give each real request enough time instead
|
||||
# of abandoning an otherwise healthy formal campaign mid-arm.
|
||||
timeout=300.0,
|
||||
max_retries=4,
|
||||
)
|
||||
|
||||
|
||||
def _sanitize_for_evidence(value):
|
||||
"""Retain raw public text while replacing large data URLs with hashes."""
|
||||
if isinstance(value, dict):
|
||||
return {key: _sanitize_for_evidence(item) for key, item in value.items()}
|
||||
if isinstance(value, list):
|
||||
return [_sanitize_for_evidence(item) for item in value]
|
||||
if isinstance(value, str) and value.startswith("data:"):
|
||||
return {
|
||||
"data_url_sha256": hashlib.sha256(value.encode("utf-8")).hexdigest(),
|
||||
"characters": len(value),
|
||||
"media_type": value.split(";", 1)[0][5:],
|
||||
}
|
||||
return value
|
||||
|
||||
|
||||
def encode_image(path: str) -> str:
|
||||
"""读取 PNG，缩放到统一宽度，编码为 data URL（base64）。"""
|
||||
img = Image.open(path).convert("RGB")
|
||||
if img.width > VISION_IMAGE_WIDTH:
|
||||
h = int(img.height * VISION_IMAGE_WIDTH / img.width)
|
||||
img = img.resize((VISION_IMAGE_WIDTH, h), Image.LANCZOS)
|
||||
buf = io.BytesIO()
|
||||
img.save(buf, format="PNG")
|
||||
b64 = base64.b64encode(buf.getvalue()).decode()
|
||||
return f"data:image/png;base64,{b64}"
|
||||
|
||||
|
||||
def _extract_json(text: str):
|
||||
"""从模型回复里稳健地抽取 JSON（容忍 ```json 代码块或前后多余文字）。"""
|
||||
text = text.strip()
|
||||
m = re.search(r"```(?:json)?\s*(.*?)```", text, re.DOTALL)
|
||||
if m:
|
||||
text = m.group(1).strip()
|
||||
# 找到第一个 { 到最后一个 }
|
||||
start = text.find("{")
|
||||
end = text.rfind("}")
|
||||
if start != -1 and end != -1:
|
||||
text = text[start:end + 1]
|
||||
return json.loads(text)
|
||||
|
||||
|
||||
def _extract_slides_md(text: str) -> str:
|
||||
"""从模型回复里抽取 slides.md 内容（容忍 ```markdown 包裹）。"""
|
||||
m = re.search(r"```(?:markdown|md)?\s*(.*?)```", text, re.DOTALL)
|
||||
if m:
|
||||
return m.group(1).strip()
|
||||
return text.strip()
|
||||
|
||||
|
||||
def _slide_count(slides: str) -> int:
|
||||
"""Count Slidev pages when the required frontmatter is present."""
|
||||
# The closing frontmatter fence plus every inter-page fence yields exactly
|
||||
# one ``\n---\n`` occurrence per rendered page.
|
||||
return slides.replace("\r\n", "\n").count("\n---\n")
|
||||
|
||||
|
||||
def _slide_contract_issues(slides: str) -> list[str]:
|
||||
"""Return deterministic density/page-count failures before rendering.
|
||||
|
||||
Vision remains the authority for pixel-level quality. This inexpensive
|
||||
source gate prevents repeatedly paying to render drafts that already
|
||||
violate the Proposer's explicit 18--20 page / four-bullet instructions.
|
||||
"""
|
||||
normalized = slides.replace("\r\n", "\n")
|
||||
pages = normalized.split("\n---\n")[1:]
|
||||
issues = []
|
||||
if not 18 <= len(pages) <= 20:
|
||||
issues.append(f"page count is {len(pages)}; required 18-20")
|
||||
for page_number, page in enumerate(pages, 1):
|
||||
bullet_count = len(re.findall(r"(?m)^\s*(?:[-*+]|[■▪•])\s+", page))
|
||||
if bullet_count > 4:
|
||||
issues.append(
|
||||
f"page {page_number} has {bullet_count} bullets; maximum is 4 total"
|
||||
)
|
||||
figure_match = re.search(r"/((?:paper_figure_)[^\s\"')>]+)", page)
|
||||
if figure_match:
|
||||
required_title = FIGURE_PAGE_TITLES.get(figure_match.group(1))
|
||||
if required_title and not re.search(
|
||||
rf"(?m)^#{{1,3}}\s+{re.escape(required_title)}\s*$", page
|
||||
):
|
||||
issues.append(
|
||||
f"page {page_number} source figure must use one-line title: "
|
||||
f"{required_title}"
|
||||
)
|
||||
style_match = re.search(r"style=[\"']([^\"']*)[\"']", page)
|
||||
style = style_match.group(1) if style_match else ""
|
||||
if not all(re.search(pattern, style) for pattern in (
|
||||
r"(?:^|;)\s*max-height:\s*460px\s*(?:;|$)",
|
||||
r"(?:^|;)\s*width:\s*100%\s*(?:;|$)",
|
||||
r"(?:^|;)\s*object-fit:\s*contain\s*(?:;|$)",
|
||||
)):
|
||||
issues.append(
|
||||
f"page {page_number} source figure must use inline style "
|
||||
"max-height: 460px; width: 100%; object-fit: contain;"
|
||||
)
|
||||
prose_lines = [
|
||||
line.strip()
|
||||
for line in page.splitlines()
|
||||
if line.strip()
|
||||
and not line.lstrip().startswith("#")
|
||||
and "<img" not in line
|
||||
]
|
||||
if bullet_count or len(prose_lines) > 1:
|
||||
issues.append(
|
||||
f"page {page_number} source figure must have only a title "
|
||||
"and at most one caption"
|
||||
)
|
||||
return issues
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# Reviewer 的审查评分标准（Proposer / 单 Agent / 独立评委共用同一套 rubric）
|
||||
# --------------------------------------------------------------------------- #
|
||||
REVIEW_RUBRIC = """你是一名严格的演示文稿质量审核员（Reviewer）。你会看到一份由 Slidev 渲染出的 PPT，
|
||||
每张图对应一页幻灯片（按顺序编号，从第 1 页开始）。请逐页检查以下问题：
|
||||
|
||||
- text_overflow（文字溢出/被裁切超出页面边界）
|
||||
- overcrowded（内容过多/过于拥挤/留白不足）
|
||||
- image_size（图片过大顶出布局，或过小看不清）
|
||||
- readability（字号过小、对比度差、代码块难读）
|
||||
- layout（对齐混乱、标题与正文比例失衡、空页）
|
||||
|
||||
请以**目标用户是听众**的严格标准审查——一页幻灯片若要点超过约 5 条、或正文文字块偏长、
|
||||
或图片挤压了文字空间，都应视为 overcrowded/image_size 问题。五个或更少的精炼要点本身
|
||||
是可接受的，不得仅因页面恰好有五个要点而报错。报告真实存在的问题，
|
||||
但不要放过"塞得太满"。对每个问题给出：page（页码，整数）、
|
||||
issue_type（上面之一）、severity（high/medium/low）、suggestion（具体、可执行的修改建议，中文）。
|
||||
|
||||
同时给出：
|
||||
- overall_score：0-100 的整体质量分（越高越好）
|
||||
- pass：布尔值，仅当整份 PPT **既无 high 也无 medium 级问题**、排版干净可读时才为 true
|
||||
|
||||
严格输出如下 JSON（不要输出任何多余文字）：
|
||||
{
|
||||
"overall_score": <int>,
|
||||
"pass": <bool>,
|
||||
"issues": [
|
||||
{"page": <int>, "issue_type": "<type>", "severity": "<high|medium|low>", "suggestion": "<中文建议>"}
|
||||
]
|
||||
}"""
|
||||
|
||||
|
||||
class Reviewer:
|
||||
"""审核者 Agent：看最新一版渲染截图，输出结构化 JSON 建议。每轮独立调用、无历史。"""
|
||||
|
||||
def __init__(self, meter: TokenMeter):
|
||||
self.client = _client()
|
||||
self.meter = meter
|
||||
|
||||
def review(self, png_paths: list[str]) -> dict:
|
||||
content = [{"type": "text",
|
||||
"text": f"这份 PPT 共 {len(png_paths)} 页，下面按页码顺序给出每一页的渲染截图。请审查。"}]
|
||||
for i, p in enumerate(png_paths, 1):
|
||||
content.append({"type": "text", "text": f"第 {i} 页："})
|
||||
content.append({"type": "image_url",
|
||||
"image_url": {"url": encode_image(p), "detail": "high"}})
|
||||
request = {
|
||||
"model": VISION_MODEL,
|
||||
"messages": [
|
||||
{"role": "system", "content": REVIEW_RUBRIC},
|
||||
{"role": "user", "content": content},
|
||||
],
|
||||
"temperature": 0.2,
|
||||
}
|
||||
started = time.monotonic()
|
||||
resp = self.client.chat.completions.create(**request)
|
||||
self.meter.add(resp, request, time.monotonic() - started)
|
||||
return _extract_json(resp.choices[0].message.content)
|
||||
|
||||
|
||||
PROPOSER_SYSTEM = """你是一名擅长把学术论文转化为演示文稿的 Proposer Agent。
|
||||
你用 Slidev 框架（Markdown + HTML）编写 PPT 源码 slides.md。
|
||||
|
||||
Slidev 语法要点：
|
||||
- 文件开头是 YAML frontmatter（--- 包裹），设置 theme: default。
|
||||
- 用单独一行的 `---`（前后空行）分隔每一页幻灯片。
|
||||
- 首页通常放标题、作者、会议。
|
||||
- 引用图片用 markdown：，可用 HTML 控制尺寸，
|
||||
例如 <img src="/speedup_bar.png" class="h-60 mx-auto" />。
|
||||
- 可用 Windi/Uno CSS 工具类控制排版（如 text-sm、grid grid-cols-2 gap-4）。
|
||||
|
||||
要求：
|
||||
- 最终生成 18-20 页，覆盖论文的标题、背景/动机、方法、实验结果、局限与结论。
|
||||
- 至少在 3 页中使用提供的图表/表格，且图文匹配。
|
||||
- 每页最多 4 个要点，不得粘贴长段原文，宁可精简也不要溢出。
|
||||
- 三张原论文图必须各占一张专用页面；为避免 UnoCSS 动态类漏编译，图片必须使用行内属性
|
||||
`style="max-height: 460px; width: 100%; object-fit: contain;"`，并且页面除标题和一行
|
||||
短图注外不得放正文，以保证整张图和图注都在页面边界内且标签可读。
|
||||
- 三张原图页面必须分别使用不会换行的精确标题：`Transformer Architecture (Figure 1)`、
|
||||
`Long-Distance Attention (Figure 3)`、`Anaphora Attention (Figure 4)`。
|
||||
- 要点统一用 Markdown `- `，不要用 `■` 等字符伪装项目符号；多个小节合计仍不得超过 4 条。
|
||||
- 只输出 slides.md 的完整内容，用 ```markdown 代码块包裹，不要额外解释。"""
|
||||
|
||||
|
||||
class Proposer:
|
||||
"""提议者 Agent：只吃文本（论文 + 累积文字反馈），产出 slides.md。"""
|
||||
|
||||
def __init__(self, meter: TokenMeter, paper_md: str, figures: dict):
|
||||
self.client = _client()
|
||||
self.meter = meter
|
||||
fig_desc = "\n".join(f"- {name}：{desc}" for name, desc in figures.items())
|
||||
first_user = (
|
||||
f"以下是论文全文（Markdown）：\n\n{paper_md}\n\n"
|
||||
f"可直接引用的图表文件（放在 Slidev public 目录，用 /文件名 引用）：\n{fig_desc}\n\n"
|
||||
f"请生成一版 18-20 页的完整初稿。内容必须忠于论文，至少引用上面列出的三张原论文图，"
|
||||
f"并覆盖问题、Transformer 架构、注意力机制、训练、主要实验、局限和结论。"
|
||||
f"不要复制长段原文；每页保持听众可读的信息密度。生成完整的 slides.md。"
|
||||
)
|
||||
# Proposer 的对话历史——只累积文本，永不加入图片
|
||||
self.messages = [
|
||||
{"role": "system", "content": PROPOSER_SYSTEM},
|
||||
{"role": "user", "content": first_user},
|
||||
]
|
||||
|
||||
def _generate(self) -> str:
|
||||
for attempt in range(3):
|
||||
request = {"model": TEXT_MODEL, "messages": self.messages, "temperature": 0.3}
|
||||
started = time.monotonic()
|
||||
resp = self.client.chat.completions.create(**request)
|
||||
self.meter.add(resp, request, time.monotonic() - started)
|
||||
reply = resp.choices[0].message.content
|
||||
self.messages.append({"role": "assistant", "content": reply})
|
||||
slides = _extract_slides_md(reply)
|
||||
issues = _slide_contract_issues(slides)
|
||||
if not issues:
|
||||
return slides
|
||||
if attempt < 2:
|
||||
self.messages.append({
|
||||
"role": "user",
|
||||
"content": (
|
||||
"硬性源码验收失败：" + "; ".join(issues) + "。"
|
||||
"请在不删除三张原论文图、不丢失主要贡献的前提下精简、合并或拆分内容，"
|
||||
"重新输出完整 slides.md；必须保持 18–20 页且每页最多 4 个 Markdown 要点。"
|
||||
),
|
||||
})
|
||||
raise RuntimeError(
|
||||
"Proposer failed the source density contract after 3 attempts: "
|
||||
+ "; ".join(issues)
|
||||
)
|
||||
|
||||
def propose(self) -> str:
|
||||
"""首轮生成。"""
|
||||
return self._generate()
|
||||
|
||||
def revise(self, review: dict) -> str:
|
||||
"""根据 Reviewer 的结构化文字反馈修订（只把 JSON 文本加入上下文）。"""
|
||||
feedback = json.dumps(review, ensure_ascii=False, indent=2)
|
||||
self.messages.append({
|
||||
"role": "user",
|
||||
"content": (
|
||||
"审核者（Reviewer）渲染了你上一版 slides.md 的每一页截图，"
|
||||
"给出如下结构化改进建议（JSON）：\n\n"
|
||||
f"{feedback}\n\n"
|
||||
"请理解这些问题并修订 slides.md（可拆页、精简文字、调整图片尺寸等），"
|
||||
"重新输出完整的 slides.md。修订后仍必须保持 18–20 页；解决拥挤时优先精简与合并，"
|
||||
"不得用无限拆页规避版面问题。"
|
||||
),
|
||||
})
|
||||
return self._generate()
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 单 Agent 自审对照组：同一段对话里既生成、又看自己的渲染图、又修订。
|
||||
# 关键区别：历次渲染的图片会**留在**同一上下文里，导致上下文随迭代快速膨胀。
|
||||
# --------------------------------------------------------------------------- #
|
||||
SELF_REVIEW_SYSTEM = PROPOSER_SYSTEM + """
|
||||
|
||||
此外，你还要**自我审查**：当收到自己 PPT 的渲染截图时，先在心里按下列标准找出问题
|
||||
（文字溢出、内容拥挤、图片尺寸、可读性、布局），再据此输出修订后的完整 slides.md。"""
|
||||
|
||||
|
||||
class SelfReviewAgent:
|
||||
"""单 Agent 自审：一条不断增长的对话，图片累积在上下文中。"""
|
||||
|
||||
def __init__(self, meter: TokenMeter, paper_md: str, figures: dict):
|
||||
self.client = _client()
|
||||
self.meter = meter
|
||||
fig_desc = "\n".join(f"- {name}：{desc}" for name, desc in figures.items())
|
||||
first_user = (
|
||||
f"以下是论文全文（Markdown）：\n\n{paper_md}\n\n"
|
||||
f"可直接引用的图表文件：\n{fig_desc}\n\n"
|
||||
f"请生成一版 18-20 页的完整初稿。内容必须忠于论文，至少引用上面列出的三张原论文图，"
|
||||
f"并覆盖问题、Transformer 架构、注意力机制、训练、主要实验、局限和结论。"
|
||||
f"不要复制长段原文；每页保持听众可读的信息密度。生成完整的 slides.md。"
|
||||
)
|
||||
self.messages = [
|
||||
{"role": "system", "content": SELF_REVIEW_SYSTEM},
|
||||
{"role": "user", "content": first_user},
|
||||
]
|
||||
|
||||
def propose(self) -> str:
|
||||
return self._generate_with_page_gate()
|
||||
|
||||
def _generate_with_page_gate(self) -> str:
|
||||
for attempt in range(3):
|
||||
request = {"model": VISION_MODEL, "messages": self.messages, "temperature": 0.3}
|
||||
started = time.monotonic()
|
||||
resp = self.client.chat.completions.create(**request)
|
||||
self.meter.add(resp, request, time.monotonic() - started)
|
||||
reply = resp.choices[0].message.content
|
||||
self.messages.append({"role": "assistant", "content": reply})
|
||||
slides = _extract_slides_md(reply)
|
||||
issues = _slide_contract_issues(slides)
|
||||
if not issues:
|
||||
return slides
|
||||
if attempt < 2:
|
||||
self.messages.append({
|
||||
"role": "user",
|
||||
"content": (
|
||||
"硬性源码验收失败：" + "; ".join(issues) + "。"
|
||||
"请保留三张原论文图和全部核心章节，压缩或重组后重新输出完整 slides.md；"
|
||||
"必须保持 18–20 页且每页最多 4 个 Markdown 要点。"
|
||||
),
|
||||
})
|
||||
raise RuntimeError(
|
||||
"Single Agent failed the source density contract after 3 attempts: "
|
||||
+ "; ".join(issues)
|
||||
)
|
||||
|
||||
def self_review_and_revise(self, png_paths: list[str]) -> str:
|
||||
"""把最新渲染截图加入**同一**上下文，让模型自审并修订。图片会一直留在历史里。"""
|
||||
content = [{"type": "text",
|
||||
"text": (f"这是你上一版 slides.md 渲染出的 {len(png_paths)} 页截图。"
|
||||
"请自我审查（文字溢出/拥挤/图片尺寸/可读性/布局），"
|
||||
"然后输出修订后的完整 slides.md。修订后硬性保持 18–20 页，"
|
||||
"并保留三张原论文图。")}]
|
||||
for i, p in enumerate(png_paths, 1):
|
||||
content.append({"type": "text", "text": f"第 {i} 页："})
|
||||
content.append({"type": "image_url",
|
||||
"image_url": {"url": encode_image(p), "detail": "high"}})
|
||||
self.messages.append({"role": "user", "content": content})
|
||||
return self._generate_with_page_gate()
|
||||
|
||||
|
||||
def independent_judge(png_paths: list[str], meter: TokenMeter) -> dict:
|
||||
"""用同一套 rubric、独立地给某一版最终 PPT 打分，用于公平比较两种方案的质量。"""
|
||||
reviewer = Reviewer(meter)
|
||||
return reviewer.review(png_paths)
|
||||
@@ -0,0 +1,640 @@
|
||||
"""
|
||||
实验 5-4：基于论文的 PPT 自动生成（提议者-审核者机制）
|
||||
|
||||
完整流程：
|
||||
1. 从精简论文（paper/sample_paper.md）+ 程序化复现的图表出发；
|
||||
2. 【双 Agent】Proposer 生成 slides.md → Slidev 渲染每页 PNG → Reviewer 用 Vision LLM
|
||||
看图给出结构化建议 → Proposer 据反馈修订 → 迭代，直到 pass 或达最大轮数；
|
||||
3. 【单 Agent 自审】同一个 Agent 生成 → 渲染 → 把自己的截图塞回**同一上下文**自审并修订 → 迭代；
|
||||
4. 用同一位“独立评委”（Vision）给两种方案的最终 PPT 打分，公平比较**质量**；
|
||||
5. 打印两种方案的**上下文 token 消耗**对比（总量、峰值单次 prompt token）。
|
||||
|
||||
运行：python demo.py # 完整对比（两种方案）
|
||||
python demo.py --help # 查看全部参数
|
||||
python demo.py --mode dual --max-rounds 1 # 快速：只跑双 Agent、只出首版
|
||||
python demo.py --smoke # 仅验证 Slidev 渲染链路，不调用任何 LLM
|
||||
python demo.py --dry-run # 离线走通提议者-审核者循环（真实渲染 + 脚本化改稿）
|
||||
依赖：Node/Slidev（渲染）、OPENAI_API_KEY（gpt-5.6-luna 视觉 + 文本；未配置时可用 OPENROUTER_API_KEY 兜底）。
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
load_dotenv()
|
||||
|
||||
import agents # noqa: E402 —— 用模块名引用 TEXT_MODEL/VISION_MODEL，便于 CLI 覆盖
|
||||
from agents import ( # noqa: E402
|
||||
Proposer, Reviewer, SelfReviewAgent, TokenMeter, independent_judge,
|
||||
)
|
||||
from make_figures import generate_all # noqa: E402
|
||||
from paper_source import PAPER, prepare_real_paper # noqa: E402
|
||||
from renderer import render_slides # noqa: E402
|
||||
|
||||
HERE = os.path.dirname(os.path.abspath(__file__))
|
||||
DEFAULT_PAPER_PATH = os.path.join(HERE, "paper", "sample_paper.md")
|
||||
DEFAULT_OUT_DIR = os.path.join(HERE, "output")
|
||||
OUT_DIR = DEFAULT_OUT_DIR # 可被 --out-dir 覆盖（main 内 global 赋值）
|
||||
MAX_ROUNDS = 3 # 每种方案的最大迭代轮数（首轮 + 最多 2 轮修订）
|
||||
|
||||
|
||||
def banner(title):
|
||||
print("\n" + "=" * 74)
|
||||
print(f" {title}")
|
||||
print("=" * 74)
|
||||
|
||||
|
||||
def save_text(name, text):
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
path = os.path.join(OUT_DIR, name)
|
||||
with open(path, "w", encoding="utf-8") as f:
|
||||
f.write(text)
|
||||
return path
|
||||
|
||||
|
||||
def retain_rendered(pngs, subdir):
|
||||
"""Copy rendered pixels into the immutable output tree used as evidence."""
|
||||
destination = os.path.join(OUT_DIR, "rendered", subdir)
|
||||
os.makedirs(destination, exist_ok=True)
|
||||
retained = []
|
||||
for source in pngs:
|
||||
target = os.path.join(destination, os.path.basename(source))
|
||||
shutil.copyfile(source, target)
|
||||
retained.append(target)
|
||||
return retained
|
||||
|
||||
|
||||
def file_sha256(path):
|
||||
digest = hashlib.sha256()
|
||||
with open(path, "rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _review_issues(review: dict) -> list:
|
||||
"""Return issue dicts; null/non-list → []; skip non-dict entries."""
|
||||
issues = review.get("issues")
|
||||
if issues is None:
|
||||
return []
|
||||
if not isinstance(issues, list):
|
||||
return []
|
||||
return [i for i in issues if isinstance(i, dict)]
|
||||
|
||||
|
||||
def summarize_review(review: dict) -> str:
|
||||
n_high = sum(1 for x in _review_issues(review) if x.get("severity") == "high")
|
||||
n_med = sum(1 for x in _review_issues(review) if x.get("severity") == "medium")
|
||||
n_low = sum(1 for x in _review_issues(review) if x.get("severity") == "low")
|
||||
return (f"score={review.get('overall_score')} pass={review.get('pass')} "
|
||||
f"issues={len(_review_issues(review))} (high={n_high}, med={n_med}, low={n_low})")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 方案 A：提议者-审核者（双 Agent）
|
||||
# --------------------------------------------------------------------------- #
|
||||
def run_proposer_reviewer(paper_md, figures, max_rounds=MAX_ROUNDS):
|
||||
banner("方案 A：提议者-审核者（双 Agent 分工）")
|
||||
proposer_meter = TokenMeter("Proposer(纯文本)")
|
||||
reviewer_meter = TokenMeter("Reviewer(每轮只看最新截图)")
|
||||
|
||||
proposer = Proposer(proposer_meter, paper_md, figures)
|
||||
reviewer = Reviewer(reviewer_meter)
|
||||
|
||||
history = [] # 每轮的 (score, review)
|
||||
slides = proposer.propose()
|
||||
final_pngs = None
|
||||
|
||||
for rnd in range(1, max_rounds + 1):
|
||||
print(f"\n[双 Agent] 第 {rnd} 轮：Proposer 产出 slides.md（{slides.count(chr(10) + '---' + chr(10)) + 1} 段分隔）")
|
||||
md_path = save_text(f"dual_round{rnd}_slides.md", slides)
|
||||
pngs = retain_rendered(
|
||||
render_slides(slides, f"dual_round{rnd}"), f"dual_round{rnd}"
|
||||
)
|
||||
final_pngs = pngs
|
||||
print(f" 渲染出 {len(pngs)} 页 PNG，例如：{pngs[0]}")
|
||||
|
||||
review = reviewer.review(pngs)
|
||||
print(f" Reviewer(Vision)审查：{summarize_review(review)}")
|
||||
# 打印真实的建议 JSON（前几条）
|
||||
print(" Reviewer 结构化建议 JSON：")
|
||||
print(_indent(json.dumps(review, ensure_ascii=False, indent=2), 4))
|
||||
save_text(f"dual_round{rnd}_review.json",
|
||||
json.dumps(review, ensure_ascii=False, indent=2))
|
||||
history.append((review.get("overall_score", 0), review))
|
||||
|
||||
blocking = [i for i in _review_issues(review)
|
||||
if i.get("severity") in ("high", "medium")]
|
||||
if review.get("pass") and not blocking:
|
||||
print(" ✓ Reviewer 判定达标（无 high/medium 问题），提前结束迭代。")
|
||||
break
|
||||
if rnd == max_rounds:
|
||||
break
|
||||
|
||||
print(" → Proposer 接收结构化文字反馈并修订（上下文只增文本，不含图片）")
|
||||
slides = proposer.revise(review)
|
||||
|
||||
return {
|
||||
"slides": slides,
|
||||
"final_pngs": final_pngs,
|
||||
"history": history,
|
||||
"proposer_meter": proposer_meter,
|
||||
"reviewer_meter": reviewer_meter,
|
||||
}
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 方案 B：单 Agent 自审
|
||||
# --------------------------------------------------------------------------- #
|
||||
def run_single_agent(paper_md, figures, max_rounds=MAX_ROUNDS):
|
||||
banner("方案 B：单 Agent 自我审查（图片累积在同一上下文）")
|
||||
meter = TokenMeter("SingleAgent(自审, 图片累积)")
|
||||
agent = SelfReviewAgent(meter, paper_md, figures)
|
||||
|
||||
slides = agent.propose()
|
||||
final_pngs = None
|
||||
|
||||
for rnd in range(1, max_rounds + 1):
|
||||
print(f"\n[单 Agent] 第 {rnd} 轮：生成/修订 slides.md")
|
||||
save_text(f"single_round{rnd}_slides.md", slides)
|
||||
pngs = retain_rendered(
|
||||
render_slides(slides, f"single_round{rnd}"), f"single_round{rnd}"
|
||||
)
|
||||
final_pngs = pngs
|
||||
print(f" 渲染出 {len(pngs)} 页 PNG")
|
||||
print(f" 当前上下文峰值 prompt token = {meter.peak_prompt_tokens}")
|
||||
|
||||
if rnd == max_rounds:
|
||||
break
|
||||
print(" → 把 %d 张截图塞回同一上下文，Agent 自审并修订（历史图片不清除）" % len(pngs))
|
||||
slides = agent.self_review_and_revise(pngs)
|
||||
|
||||
return {"slides": slides, "final_pngs": final_pngs, "meter": meter}
|
||||
|
||||
|
||||
def _indent(text, n):
|
||||
pad = " " * n
|
||||
return "\n".join(pad + line for line in text.splitlines())
|
||||
|
||||
|
||||
def smoke_test():
|
||||
"""快速冒烟：只验证 Slidev 渲染链路是否可用，不调用任何 LLM，无需 API Key。"""
|
||||
from renderer import render_slides
|
||||
banner("Smoke test：仅验证 Slidev 渲染链路（不调用 LLM）")
|
||||
demo_md = (
|
||||
"---\ntheme: default\n---\n\n"
|
||||
"# Smoke Test\n\n渲染链路自检\n\n---\n\n"
|
||||
"# 第二页\n\n- Slidev + playwright-chromium 正常\n"
|
||||
)
|
||||
pngs = render_slides(demo_md, "smoke")
|
||||
print(f"✓ 渲染成功，产出 {len(pngs)} 页 PNG：")
|
||||
for p in pngs:
|
||||
print(" ", p)
|
||||
print("Slidev 渲染链路可用。")
|
||||
|
||||
|
||||
# --------------------------------------------------------------------------- #
|
||||
# 离线 dry-run：不调用任何 LLM，走通提议者-审核者循环的**结构**。
|
||||
# - Proposer 的两版稿件是脚本化的（拥挤初稿 → 拆页修订稿），而非 LLM 生成；
|
||||
# - 渲染是**真实**的（真的调 Slidev 导出 PNG）；
|
||||
# - Reviewer 用**确定性启发式规则**（按每页文字量判定 overcrowded），
|
||||
# 明确不是 Vision LLM——仅用于离线演示“生成→渲染→审查→修订”的闭环。
|
||||
# 真实的 Vision 审查请用 `python demo.py`（需 OPENAI_API_KEY）。
|
||||
# --------------------------------------------------------------------------- #
|
||||
def _split_paragraphs(paper_md: str) -> list[str]:
|
||||
"""按空行切出正文段落，剔除标题行与表格/图片，供脚本化排版使用。"""
|
||||
paras = []
|
||||
for block in re.split(r"\n\s*\n", paper_md):
|
||||
block = block.strip()
|
||||
if not block or block.startswith("#") or block.startswith("|"):
|
||||
continue
|
||||
paras.append(re.sub(r"\s+", " ", block))
|
||||
return paras
|
||||
|
||||
|
||||
def _paper_title(paper_md: str) -> str:
|
||||
m = re.search(r"^#\s+(.+)$", paper_md, re.MULTILINE)
|
||||
return m.group(1).strip() if m else "论文演示"
|
||||
|
||||
|
||||
def _dry_first_draft(paper_md: str, figures: dict) -> str:
|
||||
"""脚本化“拥挤初稿”：把整篇论文压进约 4 页，每页塞多段原文（必然溢出）。"""
|
||||
title = _paper_title(paper_md)
|
||||
paras = _split_paragraphs(paper_md) or ["（论文正文为空）"]
|
||||
fig_names = list(figures.keys())
|
||||
# 把段落尽量塞进 3 张内容页
|
||||
groups, per = [], max(1, (len(paras) + 2) // 3)
|
||||
for i in range(0, len(paras), per):
|
||||
groups.append(paras[i:i + per])
|
||||
pages = [f"---\ntheme: default\n---\n\n# {title}\n\n自动生成演示（离线 dry-run 初稿）"]
|
||||
for gi, g in enumerate(groups[:3]):
|
||||
body = "\n\n".join(g)
|
||||
img = f"\n\n" if gi < len(fig_names) else ""
|
||||
pages.append(f"# 第 {gi + 1} 部分\n\n{body}{img}")
|
||||
return "\n\n---\n\n".join(pages) + "\n"
|
||||
|
||||
|
||||
def _dry_revised(paper_md: str, figures: dict) -> str:
|
||||
"""脚本化“修订稿”：一段一页、要点化，图表单独成页——明显更宽松，可通过启发式。"""
|
||||
title = _paper_title(paper_md)
|
||||
paras = _split_paragraphs(paper_md) or ["（论文正文为空）"]
|
||||
fig_names = list(figures.keys())
|
||||
pages = [f"---\ntheme: default\n---\n\n# {title}\n\n自动生成演示（离线 dry-run 修订稿）"]
|
||||
for i, para in enumerate(paras):
|
||||
# 每页只放一段，且截断到约 220 字，模拟“精简成要点”
|
||||
text = para if len(para) <= 220 else para[:210].rstrip() + "……"
|
||||
pages.append(f"# 要点 {i + 1}\n\n{text}")
|
||||
for name in fig_names: # 图表各自单独成页，尺寸受控
|
||||
pages.append(f"# 图表\n\n<img src=\"{name}\" class=\"h-80 mx-auto\" />")
|
||||
return "\n\n---\n\n".join(pages) + "\n"
|
||||
|
||||
|
||||
def _heuristic_review(slides_md: str) -> dict:
|
||||
"""确定性启发式（非 Vision LLM）：按每页正文字符数判定 overcrowded。"""
|
||||
parts = re.split(r"(?m)^---\s*$", slides_md)
|
||||
pages, page_no = [], 0
|
||||
for part in parts:
|
||||
s = part.strip()
|
||||
if not s or s.startswith("theme:") or "theme:" in s.split("\n")[0]:
|
||||
continue
|
||||
pages.append(s)
|
||||
issues = []
|
||||
for idx, page in enumerate(pages, 1):
|
||||
text = re.sub(r"!\[.*?\]\(.*?\)|<img[^>]*>", "", page) # 不计图片
|
||||
n = len(re.sub(r"\s+", "", text))
|
||||
if n > 500:
|
||||
issues.append({"page": idx, "issue_type": "overcrowded", "severity": "high",
|
||||
"suggestion": f"该页正文约 {n} 字，严重溢出，建议拆成多页并精简为要点。"})
|
||||
elif n > 300:
|
||||
issues.append({"page": idx, "issue_type": "overcrowded", "severity": "medium",
|
||||
"suggestion": f"该页正文约 {n} 字，偏挤，建议拆页或删减。"})
|
||||
blocking = [i for i in issues if i["severity"] in ("high", "medium")]
|
||||
score = max(0, 100 - 15 * len(blocking) - 3 * (len(issues) - len(blocking)))
|
||||
return {"overall_score": score, "pass": not blocking, "issues": issues,
|
||||
"_reviewer": "heuristic (offline, NOT a Vision LLM)"}
|
||||
|
||||
|
||||
def dry_run(paper_path: str):
|
||||
"""离线走通提议者-审核者循环：真实渲染 + 脚本化改稿 + 启发式审查。"""
|
||||
banner("Dry-run：离线演示提议者-审核者循环（真实渲染，脚本化改稿，启发式审查）")
|
||||
if not os.path.exists(paper_path):
|
||||
print(f"找不到论文文件：{paper_path}")
|
||||
sys.exit(1)
|
||||
with open(paper_path, encoding="utf-8") as f:
|
||||
paper_md = f.read()
|
||||
figures = generate_all()
|
||||
print(f"论文：{paper_path}（{len(paper_md)} 字符）；已复现图表：{', '.join(figures)}")
|
||||
print("注意：本模式不调用任何 LLM。Reviewer 由确定性启发式规则扮演（非 Vision LLM），")
|
||||
print(" 仅用于离线展示“生成→渲染→审查→修订”的闭环；真实 Vision 审查请用 `python demo.py`。")
|
||||
|
||||
stages = [
|
||||
("拥挤初稿", _dry_first_draft(paper_md, figures)),
|
||||
("拆页修订稿", _dry_revised(paper_md, figures)),
|
||||
]
|
||||
last_review = None
|
||||
for rnd, (label, slides) in enumerate(stages, 1):
|
||||
n_pages = slides.count("\n---\n") # 页分隔符数量≈页数
|
||||
print(f"\n[dry-run] 第 {rnd} 轮：Proposer 产出 slides.md（{label}，约 {n_pages} 页）")
|
||||
save_text(f"dryrun_round{rnd}_slides.md", slides)
|
||||
pngs = render_slides(slides, f"dryrun_round{rnd}")
|
||||
print(f" 渲染出 {len(pngs)} 页 PNG，例如：{pngs[0]}")
|
||||
review = _heuristic_review(slides)
|
||||
print(f" Reviewer(启发式)审查：{summarize_review(review)}")
|
||||
print(" Reviewer 结构化建议 JSON：")
|
||||
print(_indent(json.dumps(review, ensure_ascii=False, indent=2), 4))
|
||||
save_text(f"dryrun_round{rnd}_review.json",
|
||||
json.dumps(review, ensure_ascii=False, indent=2))
|
||||
last_review = review
|
||||
if review["pass"]:
|
||||
print(" ✓ Reviewer 判定达标（无 high/medium 问题），闭环结束。")
|
||||
break
|
||||
if rnd < len(stages):
|
||||
print(" → Proposer 接收结构化文字反馈并修订（拆页、精简；此处为脚本化改稿）")
|
||||
|
||||
banner("Dry-run 小结")
|
||||
print(f"闭环演示完成：初稿被判定拥挤 → 修订稿 pass={last_review['pass']}"
|
||||
f"（启发式打分 {last_review['overall_score']}）。")
|
||||
print(f"真实渲染 PNG：slidev_workspace/exports/dryrun_round*/")
|
||||
print(f"脚本化 slides.md 与审查 JSON：{OUT_DIR}/dryrun_round*")
|
||||
print("真实的 Vision 审查循环（gpt-5.6-luna 看像素）请运行：python demo.py --mode dual --max-rounds 3")
|
||||
|
||||
|
||||
def parse_args(argv=None):
|
||||
p = argparse.ArgumentParser(
|
||||
prog="demo.py",
|
||||
description="实验 5-4：论文 → PPT 自动生成（提议者-审核者 vs 单 Agent 自审对照）",
|
||||
formatter_class=argparse.RawDescriptionHelpFormatter,
|
||||
epilog=(
|
||||
"示例：\n"
|
||||
" python demo.py # 完整对比：两种方案 + 独立评委 + token 对比\n"
|
||||
" python demo.py --mode dual # 只跑双 Agent（省一半时间/费用）\n"
|
||||
" python demo.py --max-rounds 1 # 每种方案只出首版（最快的真实 LLM 冒烟）\n"
|
||||
" python demo.py --paper my.md --out-dir run1 # 换论文、换输出目录\n"
|
||||
" python demo.py --vision-model gpt-5.6-luna # 覆盖视觉模型\n"
|
||||
" python demo.py --dry-run # 离线走通提议者-审核者循环，不调用任何 LLM\n"
|
||||
" python demo.py --smoke # 仅验证 Slidev 渲染，不调用任何 LLM\n\n"
|
||||
"模型/供应商也可通过环境变量配置（见 env.example）；命令行 --text-model /\n"
|
||||
"--vision-model 优先级更高：OPENAI_API_KEY / OPENAI_BASE_URL / TEXT_MODEL / VISION_MODEL"
|
||||
),
|
||||
)
|
||||
p.add_argument("--paper", metavar="PATH", default=None,
|
||||
help="非正式兼容入口：使用本地 Markdown。省略时使用固定哈希的真实 arXiv PDF；"
|
||||
"本地 Markdown 运行永远不会通过正式实验门禁。")
|
||||
p.add_argument("--out-dir", metavar="DIR", default=DEFAULT_OUT_DIR,
|
||||
help="产物输出目录：各轮 slides.md / review.json / comparison_summary.json "
|
||||
"（默认 output/）。渲染 PNG 始终位于 slidev_workspace/exports/。")
|
||||
p.add_argument("--text-model", metavar="NAME", default=None,
|
||||
help="Proposer / 单 Agent 文本部分用的模型，覆盖 TEXT_MODEL 环境变量"
|
||||
f"（默认 {agents.TEXT_MODEL}）。")
|
||||
p.add_argument("--vision-model", metavar="NAME", default=None,
|
||||
help="Reviewer / 独立评委看图用的模型，必须支持图像输入，覆盖 VISION_MODEL "
|
||||
f"环境变量（默认 {agents.VISION_MODEL}）。")
|
||||
p.add_argument(
|
||||
"--provider", choices=["auto", "openai", "openrouter", "moonshot", "ark"],
|
||||
default="auto", help="文本与 Vision 调用的真实 API 提供商",
|
||||
)
|
||||
p.add_argument("--mode", choices=["both", "dual", "single"], default="both",
|
||||
help="运行哪种方案：both=两种都跑并对比（默认）；dual=仅提议者-审核者；"
|
||||
"single=仅单 Agent 自审。只跑一种可显著省时省钱。")
|
||||
p.add_argument("--max-rounds", type=int, default=MAX_ROUNDS, metavar="N",
|
||||
help=f"每种方案的最大迭代轮数（默认 {MAX_ROUNDS}）。设为 1 即只出首版、"
|
||||
"不修订，是最快的真实运行冒烟。")
|
||||
p.add_argument("--dry-run", action="store_true",
|
||||
help="离线演示提议者-审核者循环：真实渲染两版脚本化 slides.md（拥挤初稿→"
|
||||
"拆页修订稿），用启发式规则（非 Vision LLM）扮演 Reviewer，展示"
|
||||
"生成→渲染→审查→修订的闭环结构。不调用任何 LLM，无需 API Key。")
|
||||
p.add_argument("--smoke", action="store_true",
|
||||
help="仅验证 Slidev 渲染链路（渲染一个两页 deck），不调用任何 LLM，无需 API Key。")
|
||||
return p.parse_args(argv)
|
||||
|
||||
|
||||
def _save_partial_summary(dual, dual_final, single, single_final, source_info, args, judge_meter):
|
||||
"""单方案运行（--mode dual/single）时，落盘该方案自身的质量与 token 结果。"""
|
||||
summary = {
|
||||
"experiment": "5-4",
|
||||
"official_complete": False,
|
||||
"completion": {"campaign_complete": False, "reason": "both comparison arms are required"},
|
||||
"provider": args.provider,
|
||||
"models": {"text": agents.TEXT_MODEL, "vision": agents.VISION_MODEL},
|
||||
"source": source_info,
|
||||
"independent_judge": judge_meter.__dict__,
|
||||
}
|
||||
if dual:
|
||||
pm, rm = dual["proposer_meter"], dual["reviewer_meter"]
|
||||
summary["dual_agent"] = {
|
||||
"iteration_scores": [h[0] for h in dual["history"]],
|
||||
"final_quality": dual_final,
|
||||
"total_tokens": pm.total_tokens + rm.total_tokens,
|
||||
"peak_context_prompt_tokens": max(pm.peak_prompt_tokens, rm.peak_prompt_tokens),
|
||||
"proposer_receipts": pm.receipts,
|
||||
"reviewer_receipts": rm.receipts,
|
||||
}
|
||||
if single:
|
||||
sm = single["meter"]
|
||||
summary["single_agent"] = {
|
||||
"final_quality": single_final,
|
||||
"total_tokens": sm.total_tokens,
|
||||
"peak_context_prompt_tokens": sm.peak_prompt_tokens,
|
||||
"receipts": sm.receipts,
|
||||
}
|
||||
p = save_text("comparison_summary.json", json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
print(f"\n结果已保存：{p}")
|
||||
print(f"所有 slides.md / review.json / 渲染 PNG 位于：{OUT_DIR}/ 与 slidev_workspace/exports/")
|
||||
|
||||
|
||||
def main(argv=None):
|
||||
global OUT_DIR
|
||||
args = parse_args(argv)
|
||||
|
||||
# 输出目录（--out-dir）：所有 save_text 都写到这里
|
||||
OUT_DIR = os.path.abspath(args.out_dir)
|
||||
os.makedirs(OUT_DIR, exist_ok=True)
|
||||
os.environ["PPT_RECEIPT_CHECKPOINT"] = os.path.join(
|
||||
OUT_DIR, "receipts.checkpoint.json"
|
||||
)
|
||||
# 模型覆盖（--text-model / --vision-model 优先于环境变量）
|
||||
if args.text_model:
|
||||
agents.TEXT_MODEL = args.text_model
|
||||
if args.vision_model:
|
||||
agents.VISION_MODEL = args.vision_model
|
||||
agents.configure_provider(args.provider)
|
||||
|
||||
if args.smoke:
|
||||
smoke_test()
|
||||
return
|
||||
if args.dry_run:
|
||||
dry_run(args.paper or DEFAULT_PAPER_PATH)
|
||||
return
|
||||
if args.max_rounds < 1:
|
||||
print("--max-rounds 至少为 1")
|
||||
sys.exit(1)
|
||||
if args.paper and not os.path.exists(args.paper):
|
||||
print(f"找不到论文文件：{args.paper}（用 --paper 指定，或参考默认 paper/sample_paper.md）")
|
||||
sys.exit(1)
|
||||
provider_keys = {
|
||||
"ark": "ARK_API_KEY", "moonshot": "MOONSHOT_API_KEY",
|
||||
"openrouter": "OPENROUTER_API_KEY", "openai": "OPENAI_API_KEY",
|
||||
}
|
||||
required_key = provider_keys.get(args.provider)
|
||||
if required_key and not os.environ.get(required_key):
|
||||
print(f"请先设置 {required_key}（可参考 env.example）")
|
||||
sys.exit(1)
|
||||
if args.provider == "auto" and not any(os.environ.get(key) for key in (
|
||||
"OPENAI_API_KEY", "OPENROUTER_API_KEY"
|
||||
)):
|
||||
print("auto provider 需要 OPENAI_API_KEY 或 OPENROUTER_API_KEY")
|
||||
sys.exit(1)
|
||||
|
||||
banner("准备：固定真实论文 PDF + 原论文图")
|
||||
if args.paper:
|
||||
with open(args.paper, encoding="utf-8") as f:
|
||||
paper_md = f.read()
|
||||
figures = generate_all()
|
||||
source_info = {
|
||||
"canonical": False,
|
||||
"reason": "legacy local Markdown and programmatic figures",
|
||||
"paper_path": os.path.abspath(args.paper),
|
||||
}
|
||||
else:
|
||||
prepared = prepare_real_paper(
|
||||
OUT_DIR, os.path.join(HERE, "slidev_workspace", "public")
|
||||
)
|
||||
paper_md = prepared["paper_text"]
|
||||
figures = prepared["figures"]
|
||||
source_info = {
|
||||
"canonical": True,
|
||||
"paper": prepared["manifest"]["paper"],
|
||||
"paper_text": prepared["manifest"]["paper_text"],
|
||||
"visuals": prepared["manifest"]["visuals"],
|
||||
"manifest_path": prepared["manifest_path"],
|
||||
"pdf_path": prepared["pdf_path"],
|
||||
}
|
||||
print(f"论文：{PAPER['title']}（提取文本 {len(paper_md)} 字符）")
|
||||
print(f"输出目录：{OUT_DIR}")
|
||||
print(f"文本模型：{agents.TEXT_MODEL} 视觉模型：{agents.VISION_MODEL}")
|
||||
print(f"运行模式：{args.mode} 最大轮数：{args.max_rounds}")
|
||||
print("已生成图表：")
|
||||
for k, v in figures.items():
|
||||
print(f" {k} -> {v}")
|
||||
|
||||
# 方案 A / 方案 B（--mode 控制跑哪一种；只有 both 才能做跨方案对比）
|
||||
dual = run_proposer_reviewer(paper_md, figures, args.max_rounds) \
|
||||
if args.mode in ("both", "dual") else None
|
||||
single = run_single_agent(paper_md, figures, args.max_rounds) \
|
||||
if args.mode in ("both", "single") else None
|
||||
|
||||
# ------- 用同一位独立评委给两种方案的最终 PPT 打分（质量对比，尽量公平） -------
|
||||
banner("独立评委：对最终 PPT 打分（同一 Vision rubric）")
|
||||
judge_meter = TokenMeter("独立评委(不计入两方案成本)")
|
||||
dual_final = independent_judge(dual["final_pngs"], judge_meter) if dual else None
|
||||
single_final = independent_judge(single["final_pngs"], judge_meter) if single else None
|
||||
if dual_final:
|
||||
print(f"方案 A（双 Agent）最终质量：{summarize_review(dual_final)}")
|
||||
if single_final:
|
||||
print(f"方案 B（单 Agent）最终质量：{summarize_review(single_final)}")
|
||||
|
||||
if not (dual and single):
|
||||
# 单方案运行：跳过跨方案的 token 对比，仅落盘已有结果
|
||||
_save_partial_summary(
|
||||
dual, dual_final, single, single_final, source_info, args, judge_meter
|
||||
)
|
||||
return
|
||||
|
||||
# ------- 迭代改善情况（双 Agent） -------
|
||||
banner("迭代质量改善（方案 A：提议者-审核者）")
|
||||
scores = [h[0] for h in dual["history"]]
|
||||
if len(scores) >= 2:
|
||||
print(f"Reviewer 打分随迭代变化：{scores} "
|
||||
f"（{'↑ 改善' if scores[-1] >= scores[0] else '↓'} {scores[-1] - scores[0]:+d}）")
|
||||
else:
|
||||
print(f"仅 1 轮即达标，Reviewer 打分：{scores}")
|
||||
|
||||
# ------- 上下文 token 消耗对比 -------
|
||||
banner("上下文 Token 消耗对比：单 Agent 自审 vs 提议者-审核者")
|
||||
pm, rm, sm = dual["proposer_meter"], dual["reviewer_meter"], single["meter"]
|
||||
dual_total = pm.total_tokens + rm.total_tokens
|
||||
dual_peak = max(pm.peak_prompt_tokens, rm.peak_prompt_tokens)
|
||||
|
||||
def row(label, calls, prompt, completion, total, peak):
|
||||
print(f" {label:<34} calls={calls:<3} prompt={prompt:<8} "
|
||||
f"completion={completion:<7} total={total:<8} peak_ctx={peak}")
|
||||
|
||||
print("双 Agent（方案 A）拆分：")
|
||||
row(pm.name, pm.calls, pm.prompt_tokens, pm.completion_tokens, pm.total_tokens, pm.peak_prompt_tokens)
|
||||
row(rm.name, rm.calls, rm.prompt_tokens, rm.completion_tokens, rm.total_tokens, rm.peak_prompt_tokens)
|
||||
print("-" * 74)
|
||||
row("【方案 A 合计】", pm.calls + rm.calls, pm.prompt_tokens + rm.prompt_tokens,
|
||||
pm.completion_tokens + rm.completion_tokens, dual_total, dual_peak)
|
||||
row("【方案 B 单Agent自审】", sm.calls, sm.prompt_tokens, sm.completion_tokens,
|
||||
sm.total_tokens, sm.peak_prompt_tokens)
|
||||
print("-" * 74)
|
||||
print(f"每次调用的 prompt token 序列：")
|
||||
print(f" 方案A Proposer : {pm.per_call_prompt}")
|
||||
print(f" 方案A Reviewer : {rm.per_call_prompt} ← 每轮独立、只看最新截图，不随迭代累积")
|
||||
print(f" 方案B 单Agent : {sm.per_call_prompt} ← 图片累积在同一上下文，峰值随迭代上升")
|
||||
print()
|
||||
print(f"关键结论：")
|
||||
print(f" · 上下文峰值（单次 prompt token，决定是否撑爆上下文窗口）：")
|
||||
print(f" 方案 A = {dual_peak} 方案 B = {sm.peak_prompt_tokens} "
|
||||
f"（B/A = {sm.peak_prompt_tokens / max(dual_peak,1):.2f}x）")
|
||||
print(f" · Proposer 全程不看图片，其峰值仅 {pm.peak_prompt_tokens} token（纯文本反馈）。")
|
||||
print(f" · 方案 B 因图片在同一上下文累积，峰值最高；页数越多、迭代越多，差距越大。")
|
||||
|
||||
# 汇总落盘
|
||||
visual_names = [visual["filename"] for visual in source_info.get("visuals", [])]
|
||||
receipts = pm.receipts + rm.receipts + sm.receipts + judge_meter.receipts
|
||||
receipts_path = save_text(
|
||||
"receipts.json", json.dumps(receipts, ensure_ascii=False, indent=2)
|
||||
)
|
||||
gates = {
|
||||
"pinned_real_academic_pdf": bool(
|
||||
source_info.get("canonical")
|
||||
and source_info.get("paper", {}).get("observed_pdf_sha256") == PAPER["pdf_sha256"]
|
||||
),
|
||||
"three_original_pdf_visuals": bool(
|
||||
len(visual_names) >= 3
|
||||
and all(v["sha256"] == v["public_copy_sha256"] for v in source_info.get("visuals", []))
|
||||
),
|
||||
"both_final_decks_reference_every_source_visual": bool(
|
||||
visual_names
|
||||
and all(name in dual["slides"] and name in single["slides"] for name in visual_names)
|
||||
),
|
||||
"both_final_decks_have_10_to_20_rendered_pages": bool(
|
||||
10 <= len(dual["final_pngs"]) <= 20
|
||||
and 10 <= len(single["final_pngs"]) <= 20
|
||||
),
|
||||
"real_slidev_rendered_every_final_page": bool(
|
||||
all(os.path.exists(path) and os.path.getsize(path) > 0
|
||||
for path in dual["final_pngs"] + single["final_pngs"])
|
||||
),
|
||||
"proposer_reviewer_and_self_review_both_run": bool(
|
||||
dual["history"] and sm.calls >= 2
|
||||
),
|
||||
"same_independent_vision_judge_scored_both": bool(
|
||||
dual_final and single_final and judge_meter.calls == 2
|
||||
),
|
||||
# The manuscript acceptance criterion is about the rendered result,
|
||||
# not merely about having exercised the review mechanism. A run with
|
||||
# 10--20 pages and real Vision receipts is still incomplete when the
|
||||
# independent pixel-level judge reports blocking layout/overflow
|
||||
# defects in the proposer--reviewer deck.
|
||||
"dual_final_deck_passes_visual_acceptance": bool(
|
||||
dual_final and dual_final.get("pass") is True
|
||||
),
|
||||
"real_api_receipts_complete": bool(
|
||||
receipts
|
||||
and all(r.get("response", {}).get("id")
|
||||
and r.get("usage", {}).get("total_tokens")
|
||||
and r.get("latency_s") is not None
|
||||
for r in receipts)
|
||||
),
|
||||
}
|
||||
summary = {
|
||||
"schema_version": "2.0",
|
||||
"experiment": "5-4",
|
||||
"provider": args.provider,
|
||||
"source": source_info,
|
||||
"models": {"text": agents.TEXT_MODEL, "vision": agents.VISION_MODEL},
|
||||
"dual_agent": {
|
||||
"iteration_scores": scores,
|
||||
"final_quality": dual_final,
|
||||
"proposer_tokens": pm.__dict__,
|
||||
"reviewer_tokens": rm.__dict__,
|
||||
"total_tokens": dual_total,
|
||||
"peak_context_prompt_tokens": dual_peak,
|
||||
},
|
||||
"single_agent": {
|
||||
"final_quality": single_final,
|
||||
"tokens": sm.__dict__,
|
||||
"total_tokens": sm.total_tokens,
|
||||
"peak_context_prompt_tokens": sm.peak_prompt_tokens,
|
||||
},
|
||||
"independent_judge": judge_meter.__dict__,
|
||||
"artifact_hashes": {
|
||||
"dual_final_pngs": {os.path.basename(p): file_sha256(p) for p in dual["final_pngs"]},
|
||||
"single_final_pngs": {os.path.basename(p): file_sha256(p) for p in single["final_pngs"]},
|
||||
"raw_receipts": file_sha256(receipts_path),
|
||||
},
|
||||
"completion": {"gates": gates, "campaign_complete": all(gates.values())},
|
||||
"official_complete": all(gates.values()),
|
||||
"observed_hypothesis": {
|
||||
"dual_quality_minus_single": (
|
||||
dual_final.get("overall_score", 0) - single_final.get("overall_score", 0)
|
||||
),
|
||||
"dual_peak_context": dual_peak,
|
||||
"single_peak_context": sm.peak_prompt_tokens,
|
||||
"note": "Hypothesis outcome is reported separately from execution completion.",
|
||||
},
|
||||
}
|
||||
p = save_text("comparison_summary.json", json.dumps(summary, ensure_ascii=False, indent=2))
|
||||
print(f"\n完整对比已保存：{p}")
|
||||
print(f"所有 slides.md / review.json / 渲染 PNG 位于：{OUT_DIR}/ 与 slidev_workspace/exports/")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,14 @@
|
||||
# 必填其一：OpenAI API Key（本实验用 gpt-5.6-luna 做视觉审查 + 文本生成）
|
||||
OPENAI_API_KEY=your_openai_api_key_here
|
||||
|
||||
# 通用兜底：未配置 OPENAI_API_KEY 时自动改走 OpenRouter；
|
||||
# 默认模型 gpt-5.6-luna（gpt-5.x）直连 OpenAI 需组织实名认证，
|
||||
# 故设置了本 key 时会优先走 OpenRouter（route openai/gpt-5.6-luna）。
|
||||
# OPENROUTER_API_KEY=your_openrouter_api_key_here
|
||||
|
||||
# 可选：兼容 OpenAI 协议的自定义端点
|
||||
# OPENAI_BASE_URL=https://api.openai.com/v1
|
||||
|
||||
# 可选：指定模型（默认均为 gpt-5.6-luna；视觉模型必须支持图像输入）
|
||||
# TEXT_MODEL=gpt-5.6-luna
|
||||
# VISION_MODEL=gpt-5.6-luna
|
||||
@@ -0,0 +1,67 @@
|
||||
"""
|
||||
从论文中的数据生成两张真实的图表 PNG，放进 Slidev 的 public/ 目录，
|
||||
供 Proposer 生成的幻灯片直接引用（满足“至少 3 处原图表”的要求，同时
|
||||
让 Reviewer 的 Vision 检查能真正评估“图片尺寸是否合适”）。
|
||||
|
||||
这些图是用 matplotlib 从论文正文里的数字画出来的，属于“论文原始图表”的
|
||||
程序化复现，而非凭空捏造。
|
||||
"""
|
||||
import os
|
||||
import matplotlib
|
||||
|
||||
matplotlib.use("Agg") # 无显示环境
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
PUBLIC_DIR = os.path.join(os.path.dirname(__file__), "slidev_workspace", "public")
|
||||
|
||||
|
||||
def make_speedup_bar(path):
|
||||
"""图1：FlashAttention 相对标准实现的端到端加速比（论文第 4 节数字）。"""
|
||||
labels = ["BERT-large\n(seq 512)", "GPT-2\n(seq 1K)", "Long-Range\nArena"]
|
||||
speedups = [1.15, 3.0, 2.4]
|
||||
fig, ax = plt.subplots(figsize=(6, 3.4), dpi=150)
|
||||
bars = ax.bar(labels, speedups, color=["#4C72B0", "#DD8452", "#55A868"])
|
||||
ax.axhline(1.0, color="gray", linestyle="--", linewidth=1, label="baseline (1x)")
|
||||
ax.set_ylabel("Speedup vs. standard")
|
||||
ax.set_title("FlashAttention End-to-End Speedup")
|
||||
for b, v in zip(bars, speedups):
|
||||
ax.text(b.get_x() + b.get_width() / 2, v + 0.05, f"{v}x",
|
||||
ha="center", va="bottom", fontweight="bold")
|
||||
ax.set_ylim(0, 3.5)
|
||||
ax.legend(loc="upper left", fontsize=8)
|
||||
fig.tight_layout()
|
||||
fig.savefig(path, bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def make_memory_hierarchy(path):
|
||||
"""图2：GPU 内存层次的带宽对比（论文第 2 节表格，对数坐标）。"""
|
||||
levels = ["SRAM\n(on-chip)", "HBM\n(main GPU)", "CPU DRAM"]
|
||||
bandwidth = [19000, 1750, 12.8] # GB/s
|
||||
fig, ax = plt.subplots(figsize=(6, 3.4), dpi=150)
|
||||
bars = ax.bar(levels, bandwidth, color=["#C44E52", "#8172B3", "#937860"])
|
||||
ax.set_yscale("log")
|
||||
ax.set_ylabel("Bandwidth (GB/s, log scale)")
|
||||
ax.set_title("GPU Memory Hierarchy (A100)")
|
||||
for b, v in zip(bars, bandwidth):
|
||||
ax.text(b.get_x() + b.get_width() / 2, v * 1.15,
|
||||
f"{v:g}", ha="center", va="bottom", fontweight="bold")
|
||||
fig.tight_layout()
|
||||
fig.savefig(path, bbox_inches="tight")
|
||||
plt.close(fig)
|
||||
|
||||
|
||||
def generate_all():
|
||||
os.makedirs(PUBLIC_DIR, exist_ok=True)
|
||||
f1 = os.path.join(PUBLIC_DIR, "speedup_bar.png")
|
||||
f2 = os.path.join(PUBLIC_DIR, "memory_hierarchy.png")
|
||||
make_speedup_bar(f1)
|
||||
make_memory_hierarchy(f2)
|
||||
return {
|
||||
"/speedup_bar.png": "FlashAttention 端到端加速比柱状图（BERT 1.15x / GPT-2 3x / LRA 2.4x）",
|
||||
"/memory_hierarchy.png": "A100 GPU 内存层次带宽对比（SRAM 19TB/s / HBM ~1.75TB/s / DRAM 12.8GB/s，对数坐标）",
|
||||
}
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(generate_all())
|
||||
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"name": "paper-to-ppt-slidev",
|
||||
"private": true,
|
||||
"version": "1.0.0",
|
||||
"description": "Slidev workspace for 实验 5-4 (proposer-reviewer PPT generation)",
|
||||
"dependencies": {
|
||||
"@slidev/cli": "^0.49.29",
|
||||
"@slidev/theme-default": "^0.25.0",
|
||||
"playwright-chromium": "^1.44.0",
|
||||
"typescript": "^5.9.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,91 @@
|
||||
# FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
|
||||
|
||||
**Authors**: Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, Christopher Ré (Stanford University)
|
||||
|
||||
**Venue**: NeurIPS 2022
|
||||
|
||||
## Abstract
|
||||
|
||||
Transformers are slow and memory-hungry on long sequences, since the time and
|
||||
memory complexity of self-attention are quadratic in sequence length. We argue
|
||||
that a missing principle is making attention algorithms *IO-aware* — accounting
|
||||
for reads and writes between levels of GPU memory. We propose **FlashAttention**,
|
||||
an IO-aware exact attention algorithm that uses tiling to reduce the number of
|
||||
memory reads/writes between GPU high-bandwidth memory (HBM) and on-chip SRAM.
|
||||
FlashAttention trains Transformers faster than existing baselines: 15% end-to-end
|
||||
speedup on BERT-large, 3x speedup on GPT-2, and enables longer context, yielding
|
||||
higher quality models (0.7 better perplexity on GPT-2) and entirely new
|
||||
capabilities (the first Transformers to achieve better-than-chance performance
|
||||
on Path-X with 16K sequence length).
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
Transformer models have grown larger and deeper, but equipping them with longer
|
||||
context remains difficult, because the self-attention module at their heart has
|
||||
time and memory complexity **quadratic** in sequence length. A key question is
|
||||
whether making attention faster and more memory-efficient can help Transformer
|
||||
models address their runtime and memory challenges for long sequences.
|
||||
|
||||
Many approximate attention methods (sparse, low-rank) aim to reduce the compute
|
||||
and memory requirements. Although these methods reduce the FLOP count, they often
|
||||
do not achieve wall-clock speedup, mainly because they focus on FLOP reduction
|
||||
and ignore overheads from **memory access (IO)**.
|
||||
|
||||
Our main observation is that the principal missing ingredient is making attention
|
||||
algorithms **IO-aware** — carefully accounting for reads and writes to different
|
||||
levels of fast and slow memory (e.g., between fast on-chip SRAM and relatively
|
||||
slow HBM on a GPU).
|
||||
|
||||
## 2. Background: GPU Memory Hierarchy
|
||||
|
||||
A GPU has a memory hierarchy with different bandwidth/size trade-offs:
|
||||
|
||||
| Memory level | Size (A100) | Bandwidth |
|
||||
|--------------|-------------|-----------|
|
||||
| SRAM (on-chip) | 20 MB | 19 TB/s |
|
||||
| HBM (main GPU) | 40-80 GB | 1.5-2.0 TB/s |
|
||||
| CPU DRAM | > 1 TB | 12.8 GB/s |
|
||||
|
||||
Standard attention implementations materialize the large N x N attention matrix S
|
||||
and P to HBM, incurring O(N^2) HBM accesses. Since HBM is much slower than SRAM,
|
||||
this memory traffic dominates the runtime for long sequences.
|
||||
|
||||
## 3. Method: FlashAttention
|
||||
|
||||
FlashAttention computes exact attention with far fewer HBM accesses via two
|
||||
classical techniques adapted to attention:
|
||||
|
||||
- **Tiling**: split the inputs Q, K, V into blocks, load them from slow HBM to
|
||||
fast SRAM, compute attention per block, and accumulate the output. We never
|
||||
materialize the full N x N attention matrix in HBM.
|
||||
- **Softmax rescaling / recomputation**: the softmax normalization is computed
|
||||
incrementally across blocks using online softmax; in the backward pass, the
|
||||
attention matrix is recomputed on-chip from stored statistics rather than
|
||||
read from HBM.
|
||||
|
||||
The result is an algorithm with **O(N^2 d)** FLOPs but only **O(N^2 d^2 / M)**
|
||||
HBM accesses, where M is the SRAM size and d is the head dimension. For typical
|
||||
values, this is many times fewer HBM accesses than standard attention.
|
||||
|
||||
## 4. Experiments and Results
|
||||
|
||||
FlashAttention delivers strong end-to-end and micro-benchmark results:
|
||||
|
||||
- **BERT-large** (seq. 512): 15% faster training than the MLPerf 1.1 record.
|
||||
- **GPT-2** (seq. 1K): up to **3x** end-to-end speedup over HuggingFace and
|
||||
Megatron-LM implementations, with identical model quality.
|
||||
- **Long-range Arena** (1K-4K): 2.4x speedup vs standard attention.
|
||||
- **Long context quality**: enabling 4K context on GPT-2 gives **0.7 better
|
||||
perplexity**; a Path-X task with 16K sequence becomes solvable for the first
|
||||
time (61.4% accuracy), and block-sparse FlashAttention solves Path-256 (63.1%).
|
||||
|
||||
Memory usage scales **linearly** in sequence length (versus quadratic for
|
||||
standard attention), up to 20x memory savings at long sequence lengths.
|
||||
|
||||
## 5. Conclusion
|
||||
|
||||
By treating attention as an IO-bound problem and minimizing HBM accesses through
|
||||
tiling and recomputation, FlashAttention computes exact attention faster and with
|
||||
a smaller memory footprint than approximate methods, while remaining numerically
|
||||
exact. IO-awareness is a broadly useful principle: we hope FlashAttention
|
||||
inspires IO-aware implementations of more deep learning primitives.
|
||||
@@ -0,0 +1,156 @@
|
||||
"""Pinned real-paper preparation for Experiment 5-4.
|
||||
|
||||
The canonical campaign uses the published PDF, extracts its text directly, and
|
||||
renders three original paper figures from declared PDF page rectangles. The
|
||||
resulting manifest makes it possible to prove that the images in the Slidev
|
||||
deck came from the source PDF rather than from programmatic stand-ins.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
import fitz
|
||||
from PIL import Image
|
||||
|
||||
|
||||
PAPER = {
|
||||
"title": "Attention Is All You Need",
|
||||
"authors": "Ashish Vaswani et al.",
|
||||
"arxiv_id": "1706.03762",
|
||||
"pdf_url": "https://arxiv.org/pdf/1706.03762",
|
||||
"pdf_sha256": "bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697",
|
||||
}
|
||||
|
||||
# Coordinates are in PDF points and were registered against the pinned PDF.
|
||||
# They isolate the published figure itself (rather than surrounding body text)
|
||||
# so labels remain legible after a 16:9 slide render. Provenance does not rely
|
||||
# on pixels from the caption: the manifest records the source page, crop
|
||||
# rectangle, published figure label/caption, PDF hash, and extracted hash.
|
||||
VISUALS = [
|
||||
{
|
||||
"filename": "paper_figure_1_transformer.png",
|
||||
"pdf_page": 3,
|
||||
"source_label": "Figure 1",
|
||||
"caption": "The Transformer model architecture.",
|
||||
"rect": [92, 60, 520, 405],
|
||||
"rotation_degrees": 0,
|
||||
},
|
||||
{
|
||||
"filename": "paper_figure_3_long_distance.png",
|
||||
"pdf_page": 13,
|
||||
"source_label": "Figure 3 (long-distance dependency focus)",
|
||||
"caption": "Published encoder attention linking 'making' to 'more difficult'.",
|
||||
"rect": [190, 88, 425, 311],
|
||||
"rotation_degrees": 90,
|
||||
},
|
||||
{
|
||||
"filename": "paper_figure_4_anaphora.png",
|
||||
"pdf_page": 14,
|
||||
"source_label": "Figure 4 (lower panel, anaphora focus)",
|
||||
"caption": "Published attention from 'its' to 'Law' and 'application'.",
|
||||
"rect": [92, 360, 310, 610],
|
||||
"rotation_degrees": 90,
|
||||
},
|
||||
]
|
||||
|
||||
|
||||
def sha256(path: Path) -> str:
|
||||
digest = hashlib.sha256()
|
||||
with path.open("rb") as handle:
|
||||
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
|
||||
digest.update(chunk)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def _download(url: str, destination: Path) -> None:
|
||||
request = urllib.request.Request(url, headers={"User-Agent": "ai-agent-book/5-4"})
|
||||
with urllib.request.urlopen(request, timeout=120) as response:
|
||||
destination.write_bytes(response.read())
|
||||
|
||||
|
||||
def prepare_real_paper(run_dir: str | Path, public_dir: str | Path) -> dict:
|
||||
run_dir = Path(run_dir)
|
||||
public_dir = Path(public_dir)
|
||||
source_dir = run_dir / "source"
|
||||
visual_dir = source_dir / "source_visuals"
|
||||
source_dir.mkdir(parents=True, exist_ok=True)
|
||||
visual_dir.mkdir(parents=True, exist_ok=True)
|
||||
public_dir.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
pdf_path = source_dir / "1706.03762.pdf"
|
||||
if not pdf_path.exists():
|
||||
_download(PAPER["pdf_url"], pdf_path)
|
||||
observed_pdf_hash = sha256(pdf_path)
|
||||
if observed_pdf_hash != PAPER["pdf_sha256"]:
|
||||
raise ValueError(
|
||||
f"source PDF hash mismatch: expected {PAPER['pdf_sha256']}, "
|
||||
f"observed {observed_pdf_hash}"
|
||||
)
|
||||
|
||||
document = fitz.open(pdf_path)
|
||||
page_text = []
|
||||
for page_index, page in enumerate(document):
|
||||
page_text.append(f"\n\n## PDF page {page_index + 1}\n\n{page.get_text('text')}")
|
||||
text_path = source_dir / "paper_text.md"
|
||||
text_path.write_text(
|
||||
f"# {PAPER['title']}\n\nAuthors: {PAPER['authors']}\n" + "".join(page_text),
|
||||
encoding="utf-8",
|
||||
)
|
||||
|
||||
manifest_rows = []
|
||||
figure_descriptions = {}
|
||||
for visual in VISUALS:
|
||||
page = document[visual["pdf_page"] - 1]
|
||||
rect = fitz.Rect(visual["rect"])
|
||||
if not page.rect.contains(rect):
|
||||
raise ValueError(f"visual crop is outside page bounds: {visual}")
|
||||
pixmap = page.get_pixmap(matrix=fitz.Matrix(2.5, 2.5), clip=rect, alpha=False)
|
||||
extracted_path = visual_dir / visual["filename"]
|
||||
pixmap.save(extracted_path)
|
||||
if visual.get("rotation_degrees"):
|
||||
# The published attention labels run vertically. A lossless
|
||||
# quarter-turn makes those original pixels audience-readable on a
|
||||
# landscape slide; the transform is explicit in the manifest.
|
||||
with Image.open(extracted_path) as source_image:
|
||||
rotated = source_image.rotate(
|
||||
-int(visual["rotation_degrees"]), expand=True
|
||||
)
|
||||
rotated.save(extracted_path)
|
||||
public_path = public_dir / visual["filename"]
|
||||
shutil.copyfile(extracted_path, public_path)
|
||||
row = {
|
||||
**visual,
|
||||
"sha256": sha256(extracted_path),
|
||||
"bytes": extracted_path.stat().st_size,
|
||||
"public_copy_sha256": sha256(public_path),
|
||||
}
|
||||
manifest_rows.append(row)
|
||||
figure_descriptions[visual["filename"]] = (
|
||||
f"{visual['source_label']} from PDF page {visual['pdf_page']}: "
|
||||
f"{visual['caption']}"
|
||||
)
|
||||
document.close()
|
||||
|
||||
manifest = {
|
||||
"paper": {**PAPER, "observed_pdf_sha256": observed_pdf_hash},
|
||||
"paper_text": {
|
||||
"path": str(text_path),
|
||||
"sha256": sha256(text_path),
|
||||
"characters": len(text_path.read_text(encoding="utf-8")),
|
||||
},
|
||||
"visuals": manifest_rows,
|
||||
}
|
||||
manifest_path = visual_dir / "manifest.json"
|
||||
manifest_path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8")
|
||||
return {
|
||||
"paper_text": text_path.read_text(encoding="utf-8"),
|
||||
"figures": figure_descriptions,
|
||||
"manifest": manifest,
|
||||
"manifest_path": str(manifest_path),
|
||||
"pdf_path": str(pdf_path),
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
"""
|
||||
Slidev 渲染器：把 Proposer 生成的 slides.md 真正渲染成“每页一张 PNG”。
|
||||
|
||||
这是本实验的关键——Reviewer 之所以能看到 Proposer 看不到的“新信息”，
|
||||
正是因为我们真的把代码跑起来、渲染出了像素级的截图。
|
||||
|
||||
实现：调用本地安装的 Slidev CLI
|
||||
npx slidev export slides.md --format png --output <dir> --timeout 60000
|
||||
Slidev 的 PNG 导出底层用 playwright-chromium 打开每一页并截图。
|
||||
"""
|
||||
import glob
|
||||
import os
|
||||
import shutil
|
||||
import subprocess
|
||||
|
||||
WORKSPACE = os.path.join(os.path.dirname(__file__), "slidev_workspace")
|
||||
|
||||
|
||||
class RenderError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def render_slides(slides_md: str, out_subdir: str) -> list[str]:
|
||||
"""
|
||||
将 slides_md 文本写入工作区并导出为逐页 PNG。
|
||||
|
||||
参数:
|
||||
slides_md: 完整的 Slidev markdown 源码
|
||||
out_subdir: 输出子目录名（如 'proposer_iter1'）
|
||||
|
||||
返回:
|
||||
按页码排序的 PNG 绝对路径列表。
|
||||
"""
|
||||
os.makedirs(WORKSPACE, exist_ok=True)
|
||||
slides_path = os.path.join(WORKSPACE, "slides.md")
|
||||
with open(slides_path, "w", encoding="utf-8") as f:
|
||||
f.write(slides_md)
|
||||
|
||||
out_dir = os.path.join(WORKSPACE, "exports", out_subdir)
|
||||
if os.path.exists(out_dir):
|
||||
shutil.rmtree(out_dir)
|
||||
os.makedirs(out_dir, exist_ok=True)
|
||||
|
||||
# Slidev 的 PNG 导出会在 output 目录下生成 1.png, 2.png ...
|
||||
cmd = [
|
||||
"npx", "--no-install", "slidev", "export", "slides.md",
|
||||
"--format", "png",
|
||||
"--output", out_dir,
|
||||
"--timeout", "60000",
|
||||
"--dark", "false",
|
||||
]
|
||||
env = dict(os.environ)
|
||||
# 让 playwright 使用项目内安装的 chromium（package.json 里的 playwright-chromium）
|
||||
proc = subprocess.run(
|
||||
cmd, cwd=WORKSPACE, env=env,
|
||||
capture_output=True, text=True, timeout=600,
|
||||
)
|
||||
if proc.returncode != 0:
|
||||
raise RenderError(
|
||||
"Slidev export 失败:\n"
|
||||
f"stdout:\n{proc.stdout[-2000:]}\n"
|
||||
f"stderr:\n{proc.stderr[-2000:]}"
|
||||
)
|
||||
|
||||
pngs = sorted(
|
||||
glob.glob(os.path.join(out_dir, "*.png")),
|
||||
key=lambda p: _page_num(p),
|
||||
)
|
||||
if not pngs:
|
||||
raise RenderError(
|
||||
f"Slidev export 未产出 PNG。stdout:\n{proc.stdout[-2000:]}"
|
||||
)
|
||||
return pngs
|
||||
|
||||
|
||||
def _page_num(path: str) -> int:
|
||||
base = os.path.splitext(os.path.basename(path))[0]
|
||||
digits = "".join(ch for ch in base if ch.isdigit())
|
||||
return int(digits) if digits else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# 自检：渲染一个最小的两页 slidev
|
||||
demo = """---
|
||||
theme: default
|
||||
---
|
||||
|
||||
# Hello Slidev
|
||||
|
||||
第一页
|
||||
|
||||
---
|
||||
|
||||
# 第二页
|
||||
|
||||
- 渲染自检成功
|
||||
"""
|
||||
paths = render_slides(demo, "selftest")
|
||||
print("渲染出的 PNG：")
|
||||
for p in paths:
|
||||
print(" ", p)
|
||||
@@ -0,0 +1,5 @@
|
||||
openai>=1.30.0
|
||||
python-dotenv>=1.0
|
||||
Pillow>=10.0
|
||||
matplotlib>=3.7
|
||||
PyMuPDF>=1.25.0
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Null review issues must summarize without TypeError."""
|
||||
import sys
|
||||
from unittest.mock import MagicMock
|
||||
|
||||
# demo.py imports heavy optional deps at module import time.
|
||||
sys.modules.setdefault("dotenv", MagicMock())
|
||||
sys.modules.setdefault("agents", MagicMock())
|
||||
sys.modules.setdefault("make_figures", MagicMock())
|
||||
sys.modules.setdefault("renderer", MagicMock())
|
||||
|
||||
from demo import summarize_review, _review_issues
|
||||
|
||||
|
||||
def test_null_issues_like_empty():
|
||||
assert _review_issues({"issues": None}) == []
|
||||
text = summarize_review({"overall_score": 90, "pass": True, "issues": None})
|
||||
assert "issues=0" in text
|
||||
assert "high=0" in text
|
||||
|
||||
|
||||
def test_issues_preserved():
|
||||
issues = [{"severity": "high"}]
|
||||
assert _review_issues({"issues": issues}) == issues
|
||||
text = summarize_review({"overall_score": 50, "pass": False, "issues": issues})
|
||||
assert "high=1" in text
|
||||
|
||||
|
||||
def test_non_dict_issue_items_dropped():
|
||||
assert _review_issues({"issues": [None, {"severity": "high"}, "x"]}) == [{"severity": "high"}]
|
||||
text = summarize_review({"overall_score": 40, "pass": False, "issues": [None, {"severity": "high"}]})
|
||||
assert "high=1" in text
|
||||
assert "issues=1" in text
|
||||
@@ -0,0 +1,71 @@
|
||||
import importlib.util
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
spec = importlib.util.spec_from_file_location("paper_to_ppt_agents_real", Path(__file__).with_name("agents.py"))
|
||||
agents = importlib.util.module_from_spec(spec)
|
||||
assert spec.loader is not None
|
||||
spec.loader.exec_module(agents)
|
||||
_slide_count = agents._slide_count
|
||||
_slide_contract_issues = agents._slide_contract_issues
|
||||
|
||||
|
||||
def deck(pages: int) -> str:
|
||||
body = ["---\ntheme: default\n---\n\n# Page 1"]
|
||||
body.extend(f"# Page {index}" for index in range(2, pages + 1))
|
||||
return "\n\n---\n\n".join(body)
|
||||
|
||||
|
||||
def test_slide_count_matches_slidev_frontmatter_and_separators():
|
||||
assert _slide_count(deck(10)) == 10
|
||||
assert _slide_count(deck(20)) == 20
|
||||
assert _slide_count(deck(22)) == 22
|
||||
|
||||
|
||||
def test_source_contract_requires_18_to_20_pages():
|
||||
assert _slide_contract_issues(deck(18)) == []
|
||||
assert _slide_contract_issues(deck(20)) == []
|
||||
assert "page count is 17" in _slide_contract_issues(deck(17))[0]
|
||||
|
||||
|
||||
def test_source_contract_caps_total_bullets_per_page():
|
||||
slides = deck(18) + "\n\n- one\n- two\n- three\n- four\n- five\n"
|
||||
issues = _slide_contract_issues(slides)
|
||||
assert issues == ["page 18 has 5 bullets; maximum is 4 total"]
|
||||
|
||||
|
||||
def test_source_contract_counts_non_markdown_fake_bullets():
|
||||
slides = deck(18) + "\n\n■ one\n▪ two\n• three\n- four\n- five\n"
|
||||
assert _slide_contract_issues(slides) == [
|
||||
"page 18 has 5 bullets; maximum is 4 total"
|
||||
]
|
||||
|
||||
|
||||
def test_source_contract_enforces_dedicated_bounded_figure_page():
|
||||
valid = deck(18) + (
|
||||
'\n\n## Long-Distance Attention (Figure 3)\n\n'
|
||||
'<img src="/paper_figure_3_long_distance.png" '
|
||||
'style="max-height: 460px; width: 100%; object-fit: contain;" />'
|
||||
'\n\nOne short caption.\n'
|
||||
)
|
||||
assert _slide_contract_issues(valid) == []
|
||||
|
||||
oversized = valid.replace("max-height: 460px", "max-height: 650px")
|
||||
assert _slide_contract_issues(oversized) == [
|
||||
"page 18 source figure must use inline style max-height: 460px; "
|
||||
"width: 100%; object-fit: contain;"
|
||||
]
|
||||
|
||||
crowded = valid + "\n- extra point\n"
|
||||
assert _slide_contract_issues(crowded) == [
|
||||
"page 18 source figure must have only a title and at most one caption"
|
||||
]
|
||||
|
||||
wrapped_title = valid.replace(
|
||||
"## Long-Distance Attention (Figure 3)",
|
||||
"## Attention Visualization: Long-Distance Dependencies",
|
||||
)
|
||||
assert _slide_contract_issues(wrapped_title) == [
|
||||
"page 18 source figure must use one-line title: "
|
||||
"Long-Distance Attention (Figure 3)"
|
||||
]
|
||||
+90
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"overall_score": 65,
|
||||
"pass": false,
|
||||
"issues": [
|
||||
{
|
||||
"page": 2,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将6个要点拆分为2页，每页不超过3-4个要点，或精简合并相似内容"
|
||||
},
|
||||
{
|
||||
"page": 3,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将RNN和卷积方法分两页展示，或每个部分保留2个核心要点"
|
||||
},
|
||||
{
|
||||
"page": 5,
|
||||
"issue_type": "image_size",
|
||||
"severity": "high",
|
||||
"suggestion": "放大架构图至至少占页面60%空间，确保图中文字和模块清晰可见"
|
||||
},
|
||||
{
|
||||
"page": 5,
|
||||
"issue_type": "readability",
|
||||
"severity": "medium",
|
||||
"suggestion": "删除图片下方的小字体说明文字，或单独用一页解释架构细节"
|
||||
},
|
||||
{
|
||||
"page": 7,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将3个子要点拆分为独立要点，或移至下一页单独讲解decoder子层结构"
|
||||
},
|
||||
{
|
||||
"page": 9,
|
||||
"issue_type": "image_size",
|
||||
"severity": "high",
|
||||
"suggestion": "放大多头注意力机制图，确保每个头的结构和连接关系清晰可辨"
|
||||
},
|
||||
{
|
||||
"page": 9,
|
||||
"issue_type": "readability",
|
||||
"severity": "medium",
|
||||
"suggestion": "删除图片下方的小字体说明文字，公式保持现有大小即可"
|
||||
},
|
||||
{
|
||||
"page": 13,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将Data & Batching和Hardware & Schedule分两页展示，或各保留2个最关键要点"
|
||||
},
|
||||
{
|
||||
"page": 15,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "表格保留核心对比数据，4个要点精简为2-3个核心结论"
|
||||
},
|
||||
{
|
||||
"page": 18,
|
||||
"issue_type": "image_size",
|
||||
"severity": "high",
|
||||
"suggestion": "单独一页放大展示注意力可视化图，确保能看清词语间的注意力连接关系"
|
||||
},
|
||||
{
|
||||
"page": 18,
|
||||
"issue_type": "readability",
|
||||
"severity": "medium",
|
||||
"suggestion": "删除图片下方的小字体说明文字，用简洁标题概括可视化内容"
|
||||
},
|
||||
{
|
||||
"page": 19,
|
||||
"issue_type": "image_size",
|
||||
"severity": "high",
|
||||
"suggestion": "单独一页放大展示指代消解可视化图，确保紫色连接线和词语清晰可见"
|
||||
},
|
||||
{
|
||||
"page": 19,
|
||||
"issue_type": "readability",
|
||||
"severity": "medium",
|
||||
"suggestion": "删除图片下方的小字体说明文字，直接在标题中说明可视化内容"
|
||||
},
|
||||
{
|
||||
"page": 21,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将Key Contributions和Future Directions分两页展示，每部分保留3-4个要点"
|
||||
}
|
||||
]
|
||||
}
|
||||
+261
@@ -0,0 +1,261 @@
|
||||
---
|
||||
theme: default
|
||||
---
|
||||
|
||||
# Attention Is All You Need
|
||||
## A Revolutionary Architecture for Sequence Transduction
|
||||
|
||||
Ashish Vaswani et al.
|
||||
NIPS 2017
|
||||
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
- Proposes **Transformer** - first model based solely on attention mechanisms
|
||||
- Dispenses with recurrence and convolutions entirely
|
||||
- Superior quality while being more parallelizable and requiring less training time
|
||||
- Achieves 28.4 BLEU on WMT 2014 English-to-German (↑2 BLEU over previous best)
|
||||
- Achieves 41.8 BLEU on WMT 2014 English-to-French (new state-of-the-art)
|
||||
- Generalizes well to other tasks like English constituency parsing
|
||||
|
||||
---
|
||||
|
||||
## Background: The Problem with Existing Approaches
|
||||
|
||||
### Recurrent Neural Networks (RNNs/LSTMs/GRUs)
|
||||
- Inherently sequential computation
|
||||
- Cannot parallelize within training examples
|
||||
- Difficult to learn long-range dependencies
|
||||
|
||||
### Convolutional Approaches
|
||||
- ByteNet, ConvS2S use CNNs for parallelization
|
||||
- Number of operations grows with distance between positions
|
||||
- Linear (ConvS2S) or logarithmic (ByteNet) path lengths
|
||||
|
||||
---
|
||||
|
||||
## Key Insight: Attention is Sufficient
|
||||
|
||||
Attention mechanisms allow modeling dependencies without regard to distance, but were previously used with RNNs.
|
||||
|
||||
**Transformer**: First transduction model relying entirely on self-attention to compute representations without:
|
||||
- Sequence-aligned RNNs
|
||||
- Convolutions
|
||||
|
||||
---
|
||||
|
||||
## Transformer Architecture Overview
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-80 mx-auto" />
|
||||
|
||||
*Encoder-decoder structure with stacked self-attention and feed-forward layers*
|
||||
|
||||
---
|
||||
|
||||
## Encoder Structure
|
||||
|
||||
- Stack of **6 identical layers**
|
||||
- Each layer has two sub-layers:
|
||||
1. **Multi-head self-attention** mechanism
|
||||
2. **Position-wise fully connected feed-forward network**
|
||||
- Residual connections around each sub-layer
|
||||
- Layer normalization after each sub-layer
|
||||
- All sub-layers produce outputs of dimension `d_model = 512`
|
||||
|
||||
---
|
||||
|
||||
## Decoder Structure
|
||||
|
||||
- Stack of **6 identical layers**
|
||||
- Three sub-layers per layer:
|
||||
1. Masked multi-head self-attention (prevents leftward information flow)
|
||||
2. Multi-head attention over encoder output
|
||||
3. Position-wise fully connected feed-forward network
|
||||
- Residual connections and layer normalization
|
||||
- Output embeddings offset by one position (auto-regressive property)
|
||||
|
||||
---
|
||||
|
||||
## Attention Mechanism
|
||||
|
||||
### Scaled Dot-Product Attention
|
||||
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
|
||||
|
||||
- $Q$ (queries), $K$ (keys), $V$ (values) are matrices
|
||||
- Scaling by $\frac{1}{\sqrt{d_k}}$ prevents gradients from becoming too small
|
||||
- Faster and more space-efficient than additive attention
|
||||
|
||||
---
|
||||
|
||||
## Multi-Head Attention
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-40 mx-auto" />
|
||||
|
||||
- Projects queries, keys, values $h$ times with different learned projections
|
||||
- Performs attention in parallel on projected versions
|
||||
- Concatenates results and projects again
|
||||
|
||||
$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O$$
|
||||
where $\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$
|
||||
|
||||
---
|
||||
|
||||
## Three Applications of Attention
|
||||
|
||||
1. **Encoder-decoder attention**: Queries from decoder, keys/values from encoder
|
||||
2. **Encoder self-attention**: All keys, values, queries from previous encoder layer
|
||||
3. **Decoder self-attention**: All positions in decoder up to current position (masked)
|
||||
|
||||
---
|
||||
|
||||
## Positional Encoding
|
||||
|
||||
Since model has no recurrence/convolution, we inject positional information:
|
||||
|
||||
$$\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
|
||||
$$\text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
|
||||
|
||||
- Same dimension as embeddings ($d_{\text{model}}$)
|
||||
- Allows model to learn relative position information
|
||||
- Performed nearly as well as learned positional embeddings
|
||||
|
||||
---
|
||||
|
||||
## Why Self-Attention?
|
||||
|
||||
| Layer Type | Complexity | Sequential Operations | Max Path Length |
|
||||
|------------|------------|-----------------------|-----------------|
|
||||
| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
|
||||
| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
|
||||
| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
|
||||
|
||||
- Constant path length between any positions
|
||||
- More parallelizable than RNNs
|
||||
- Better computational efficiency for typical sentence lengths
|
||||
|
||||
---
|
||||
|
||||
## Training Details
|
||||
|
||||
### Data & Batching
|
||||
- WMT 2014 English-German (4.5M sentence pairs)
|
||||
- WMT 2014 English-French (36M sentence pairs)
|
||||
- Byte-pair encoding (37K shared vocab for EN-DE)
|
||||
- Batches with ~25000 source and target tokens
|
||||
|
||||
### Hardware & Schedule
|
||||
- 8 NVIDIA P100 GPUs
|
||||
- Base model: 100,000 steps (12 hours)
|
||||
- Big model: 300,000 steps (3.5 days)
|
||||
|
||||
---
|
||||
|
||||
## Training Details (Cont.)
|
||||
|
||||
### Optimizer
|
||||
- Adam with $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 10^{-9}$
|
||||
- Learning rate schedule:
|
||||
$$\text{lrate} = d_{\text{model}}^{-0.5} \cdot \min(\text{step\_num}^{-0.5}, \text{step\_num} \cdot \text{warmup\_steps}^{-1.5})$$
|
||||
- Warmup steps = 4000
|
||||
|
||||
### Regularization
|
||||
- Residual dropout (P_drop = 0.1)
|
||||
- Label smoothing ($\epsilon_{ls} = 0.1$)
|
||||
|
||||
---
|
||||
|
||||
## Machine Translation Results
|
||||
|
||||
| Model | EN-DE BLEU | EN-FR BLEU | Training Cost (FLOPs) |
|
||||
|-------|------------|------------|-----------------------|
|
||||
| GNMT + RL Ensemble | 26.30 | 41.16 | $1.8 \cdot 10^{20}$ |
|
||||
| ConvS2S Ensemble | 26.36 | 41.29 | $7.7 \cdot 10^{19}$ |
|
||||
| **Transformer (big)** | **28.4** | **41.8** | **$2.3 \cdot 10^{19}$** |
|
||||
|
||||
- Transformer outperforms all previous state-of-the-art models
|
||||
- Achieves better results with significantly lower training cost
|
||||
- 28.4 BLEU on EN-DE (↑2 BLEU over previous best)
|
||||
- 41.8 BLEU on EN-FR (new state-of-the-art)
|
||||
|
||||
---
|
||||
|
||||
## Model Variations Analysis
|
||||
|
||||
| Variation | Dev PPL | Dev BLEU |
|
||||
|-----------|---------|----------|
|
||||
| Base model | 4.92 | 25.8 |
|
||||
| Single attention head | 5.29 | 24.9 |
|
||||
| No dropout | 5.77 | 24.6 |
|
||||
| Learned positional embeddings | 4.92 | 25.7 |
|
||||
| Big model | 4.33 | 26.4 |
|
||||
|
||||
- Multiple attention heads improve performance
|
||||
- Dropout is crucial for avoiding overfitting
|
||||
- Sinusoidal and learned positional encodings perform similarly
|
||||
- Larger models (more dimensions, more heads) improve performance
|
||||
|
||||
---
|
||||
|
||||
## Generalization to Other Tasks: English Constituency Parsing
|
||||
|
||||
| Parser | Training | WSJ 23 F1 |
|
||||
|--------|----------|-----------|
|
||||
| Petrov et al. (2006) | WSJ only | 90.4 |
|
||||
| Dyer et al. (2016) | WSJ only | 91.7 |
|
||||
| **Transformer (4 layers)** | **WSJ only** | **91.3** |
|
||||
| Vinyals & Kaiser et al. | Semi-supervised | 92.1 |
|
||||
| **Transformer (4 layers)** | **Semi-supervised** | **92.7** |
|
||||
|
||||
- Transformer performs well despite no task-specific tuning
|
||||
- Outperforms previous models in semi-supervised setting
|
||||
- Shows generalization ability beyond machine translation
|
||||
|
||||
---
|
||||
|
||||
## Attention Visualization: Long-Distance Dependencies
|
||||
|
||||
<img src="/paper_figure_3_long_distance.png" class="h-70 mx-auto" />
|
||||
|
||||
*Encoder self-attention in layer 5 showing attention to distant dependency of the verb "making"*
|
||||
|
||||
---
|
||||
|
||||
## Attention Visualization: Anaphora Resolution
|
||||
|
||||
<img src="/paper_figure_4_anaphora.png" class="h-70 mx-auto" />
|
||||
|
||||
*Attention heads involved in resolving "its" reference to "The Law"*
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- Computational complexity grows quadratically with sequence length
|
||||
- Less effective for very long sequences (e.g., books, articles)
|
||||
- Still requires sequential generation in decoder
|
||||
- Limited ability to model hierarchical structure compared to some syntactic models
|
||||
|
||||
---
|
||||
|
||||
## Conclusion and Future Work
|
||||
|
||||
### Key Contributions
|
||||
- Introduced Transformer architecture based solely on attention
|
||||
- Achieved new state-of-the-art results in machine translation
|
||||
- Demonstrated improved parallelization and reduced training time
|
||||
- Showed generalization to other tasks like constituency parsing
|
||||
|
||||
### Future Directions
|
||||
- Apply to other modalities (images, audio, video)
|
||||
- Investigate local, restricted attention for large inputs
|
||||
- Make generation less sequential
|
||||
- Explore interpretability of attention mechanisms
|
||||
|
||||
---
|
||||
|
||||
## Thank You
|
||||
|
||||
Code available at: https://github.com/tensorflow/tensor2tensor
|
||||
|
||||
arXiv:1706.03762v7 [cs.CL]
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
{
|
||||
"overall_score": 65,
|
||||
"pass": false,
|
||||
"issues": [
|
||||
{
|
||||
"page": 7,
|
||||
"issue_type": "image_size",
|
||||
"severity": "medium",
|
||||
"suggestion": "缩小架构图尺寸，确保下方说明文字有足够空间，避免图片挤压文字区域"
|
||||
},
|
||||
{
|
||||
"page": 8,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将5个要点拆分为两页，建议将\"每个层包含两个子层\"及其子要点单独成页"
|
||||
},
|
||||
{
|
||||
"page": 12,
|
||||
"issue_type": "image_size",
|
||||
"severity": "medium",
|
||||
"suggestion": "调整多头注意力架构图大小，确保图片下方的文字说明完整可见且易于阅读"
|
||||
},
|
||||
{
|
||||
"page": 13,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "low",
|
||||
"suggestion": "将三个注意力应用拆分为单独页面，或简化每个应用的子要点描述"
|
||||
},
|
||||
{
|
||||
"page": 21,
|
||||
"issue_type": "image_size",
|
||||
"severity": "high",
|
||||
"suggestion": "大幅放大注意力可视化图，确保图中文字和颜色编码清晰可辨，必要时单独成页展示"
|
||||
},
|
||||
{
|
||||
"page": 22,
|
||||
"issue_type": "image_size",
|
||||
"severity": "high",
|
||||
"suggestion": "放大指代消解可视化图，增加图片分辨率，确保线条和文字清晰可读"
|
||||
}
|
||||
]
|
||||
}
|
||||
+256
@@ -0,0 +1,256 @@
|
||||
---
|
||||
theme: default
|
||||
---
|
||||
|
||||
# Attention Is All You Need
|
||||
## A Revolutionary Architecture for Sequence Transduction
|
||||
|
||||
Ashish Vaswani et al.
|
||||
NIPS 2017
|
||||
|
||||
---
|
||||
|
||||
## Abstract: Key Innovation
|
||||
|
||||
- Proposes **Transformer** - first model based solely on attention mechanisms
|
||||
- Dispenses with recurrence and convolutions entirely
|
||||
- More parallelizable and requires significantly less training time
|
||||
|
||||
---
|
||||
|
||||
## Abstract: Performance Highlights
|
||||
|
||||
- Achieves 28.4 BLEU on WMT 2014 English-to-German
|
||||
- Improves over existing best results by over 2 BLEU
|
||||
- Establishes new state-of-the-art 41.8 BLEU on WMT 2014 English-to-French
|
||||
- Generalizes well to other tasks like English constituency parsing
|
||||
|
||||
---
|
||||
|
||||
## Background: Limitations of RNNs
|
||||
|
||||
### Recurrent Neural Networks (RNNs/LSTMs/GRUs)
|
||||
- Inherently sequential computation
|
||||
- Cannot parallelize within training examples
|
||||
- Difficult to learn long-range dependencies
|
||||
- Memory constraints limit batching for long sequences
|
||||
|
||||
---
|
||||
|
||||
## Background: Limitations of Convolutional Approaches
|
||||
|
||||
### CNN-based Models (ByteNet, ConvS2S)
|
||||
- Use convolutions for parallelization
|
||||
- Number of operations grows with distance between positions
|
||||
- Linear growth for ConvS2S
|
||||
- Logarithmic growth for ByteNet
|
||||
- Longer path lengths between distant positions
|
||||
|
||||
---
|
||||
|
||||
## Key Insight: Attention is Sufficient
|
||||
|
||||
Attention mechanisms allow modeling dependencies without regard to distance, but were previously used with RNNs.
|
||||
|
||||
**Transformer**: First transduction model relying entirely on self-attention to compute representations without:
|
||||
- Sequence-aligned RNNs
|
||||
- Convolutions
|
||||
|
||||
---
|
||||
|
||||
## Transformer Architecture Overview
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-96 mx-auto" />
|
||||
|
||||
---
|
||||
|
||||
## Encoder Structure
|
||||
|
||||
- Stack of **6 identical layers**
|
||||
- Each layer contains two sub-layers:
|
||||
1. Multi-head self-attention mechanism
|
||||
2. Position-wise fully connected feed-forward network
|
||||
- Residual connections around each sub-layer
|
||||
- Layer normalization after each sub-layer
|
||||
|
||||
---
|
||||
|
||||
## Decoder Structure
|
||||
|
||||
- Stack of **6 identical layers**
|
||||
- Three sub-layers per layer:
|
||||
1. Masked multi-head self-attention
|
||||
2. Multi-head attention over encoder output
|
||||
3. Position-wise fully connected feed-forward network
|
||||
- Residual connections and layer normalization
|
||||
|
||||
---
|
||||
|
||||
## Decoder: Masking Mechanism
|
||||
|
||||
- Prevents positions from attending to subsequent positions
|
||||
- Ensures predictions for position i depend only on:
|
||||
- Known outputs at positions < i
|
||||
- Output embeddings offset by one position
|
||||
- Preserves auto-regressive property
|
||||
|
||||
---
|
||||
|
||||
## Attention Mechanism
|
||||
|
||||
### Scaled Dot-Product Attention
|
||||
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
|
||||
|
||||
- $Q$ (queries), $K$ (keys), $V$ (values) are matrices
|
||||
- Scaling by $\frac{1}{\sqrt{d_k}}$ prevents gradients from becoming too small
|
||||
- Faster and more space-efficient than additive attention
|
||||
|
||||
---
|
||||
|
||||
## Multi-Head Attention
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-80 mx-auto" />
|
||||
|
||||
- Projects queries, keys, values $h$ times with different learned projections
|
||||
- Performs attention in parallel on projected versions
|
||||
- Concatenates results and projects again
|
||||
|
||||
---
|
||||
|
||||
## Three Applications of Attention
|
||||
|
||||
1. **Encoder-decoder attention**:
|
||||
- Queries from decoder
|
||||
- Keys/values from encoder output
|
||||
|
||||
2. **Encoder self-attention**:
|
||||
- All keys, values, queries from previous encoder layer
|
||||
- Each position attends to all positions
|
||||
|
||||
3. **Decoder self-attention**:
|
||||
- All positions in decoder up to current position
|
||||
- Masked to prevent future information flow
|
||||
|
||||
---
|
||||
|
||||
## Positional Encoding
|
||||
|
||||
Since model has no recurrence/convolution, we inject positional information:
|
||||
|
||||
$$\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
|
||||
$$\text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
|
||||
|
||||
- Same dimension as embeddings ($d_{\text{model}}$)
|
||||
- Allows model to learn relative position information
|
||||
- Performed nearly as well as learned positional embeddings
|
||||
|
||||
---
|
||||
|
||||
## Why Self-Attention?
|
||||
|
||||
| Layer Type | Complexity | Sequential Operations | Max Path Length |
|
||||
|------------|------------|-----------------------|-----------------|
|
||||
| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
|
||||
| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
|
||||
| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
|
||||
|
||||
---
|
||||
|
||||
## Training: Data & Batching
|
||||
|
||||
- WMT 2014 English-German (4.5M sentence pairs)
|
||||
- WMT 2014 English-French (36M sentence pairs)
|
||||
- Byte-pair encoding (37K shared vocab for EN-DE)
|
||||
- Batches with ~25000 source and target tokens
|
||||
|
||||
---
|
||||
|
||||
## Training: Hardware & Schedule
|
||||
|
||||
- 8 NVIDIA P100 GPUs
|
||||
- Base model: 100,000 steps (12 hours)
|
||||
- Big model: 300,000 steps (3.5 days)
|
||||
- Adam optimizer with scheduled learning rate
|
||||
|
||||
---
|
||||
|
||||
## Machine Translation Results
|
||||
|
||||
| Model | EN-DE BLEU | EN-FR BLEU | Training Cost (FLOPs) |
|
||||
|-------|------------|------------|-----------------------|
|
||||
| GNMT + RL Ensemble | 26.30 | 41.16 | $1.8 \cdot 10^{20}$ |
|
||||
| ConvS2S Ensemble | 26.36 | 41.29 | $7.7 \cdot 10^{19}$ |
|
||||
| **Transformer (big)** | **28.4** | **41.8** | **$2.3 \cdot 10^{19}$** |
|
||||
|
||||
---
|
||||
|
||||
## Model Variations Analysis
|
||||
|
||||
| Variation | Dev PPL | Dev BLEU |
|
||||
|-----------|---------|----------|
|
||||
| Base model | 4.92 | 25.8 |
|
||||
| Single attention head | 5.29 | 24.9 |
|
||||
| No dropout | 5.77 | 24.6 |
|
||||
| Learned positional embeddings | 4.92 | 25.7 |
|
||||
|
||||
---
|
||||
|
||||
## Generalization to Constituency Parsing
|
||||
|
||||
| Parser | Training | WSJ 23 F1 |
|
||||
|--------|----------|-----------|
|
||||
| Previous state-of-the-art | WSJ only | 91.7 |
|
||||
| **Transformer (4 layers)** | **WSJ only** | **91.3** |
|
||||
| Previous state-of-the-art | Semi-supervised | 92.1 |
|
||||
| **Transformer (4 layers)** | **Semi-supervised** | **92.7** |
|
||||
|
||||
---
|
||||
|
||||
## Attention Visualization: Long-Distance Dependencies
|
||||
|
||||
<img src="/paper_figure_3_long_distance.png" class="h-96 mx-auto" />
|
||||
|
||||
*Encoder self-attention showing long-distance dependency for "making"*
|
||||
|
||||
---
|
||||
|
||||
## Attention Visualization: Anaphora Resolution
|
||||
|
||||
<img src="/paper_figure_4_anaphora.png" class="h-96 mx-auto" />
|
||||
|
||||
*Attention heads resolving "its" reference to "The Law"*
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- Computational complexity grows quadratically with sequence length
|
||||
- Less effective for very long sequences
|
||||
- Still requires sequential generation in decoder
|
||||
- Limited ability to model hierarchical structure
|
||||
|
||||
---
|
||||
|
||||
## Key Contributions
|
||||
|
||||
- Introduced Transformer architecture based solely on attention
|
||||
- Achieved new state-of-the-art results in machine translation
|
||||
- Demonstrated improved parallelization and reduced training time
|
||||
- Showed generalization to other tasks like constituency parsing
|
||||
|
||||
---
|
||||
|
||||
## Future Work
|
||||
|
||||
- Apply to other modalities (images, audio, video)
|
||||
- Investigate local, restricted attention for large inputs
|
||||
- Make generation less sequential
|
||||
- Explore interpretability of attention mechanisms
|
||||
|
||||
---
|
||||
|
||||
## Thank You
|
||||
|
||||
Code available at: https://github.com/tensorflow/tensor2tensor
|
||||
|
||||
arXiv:1706.03762v7 [cs.CL]
|
||||
+18
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"overall_score": 85,
|
||||
"pass": false,
|
||||
"issues": [
|
||||
{
|
||||
"page": 23,
|
||||
"issue_type": "readability",
|
||||
"severity": "medium",
|
||||
"suggestion": "原图彩色注意力可视化转为黑白后对比度差，不同注意力头难以区分。建议添加灰度区分或图案填充来标识不同注意力头，或增加图例说明关键颜色对应的含义。"
|
||||
},
|
||||
{
|
||||
"page": 24,
|
||||
"issue_type": "readability",
|
||||
"severity": "medium",
|
||||
"suggestion": "原图彩色注意力可视化转为黑白后线条区分度低。建议使用不同线型（实线/虚线/点线）或线宽来区分不同注意力头，确保黑白打印下仍能清晰识别关键依赖关系。"
|
||||
}
|
||||
]
|
||||
}
|
||||
+273
@@ -0,0 +1,273 @@
|
||||
---
|
||||
theme: default
|
||||
---
|
||||
|
||||
# Attention Is All You Need
|
||||
## A Revolutionary Architecture for Sequence Transduction
|
||||
|
||||
Ashish Vaswani et al.
|
||||
NIPS 2017
|
||||
|
||||
---
|
||||
|
||||
## Abstract: Key Innovation
|
||||
|
||||
- Proposes **Transformer** - first model based solely on attention mechanisms
|
||||
- Dispenses with recurrence and convolutions entirely
|
||||
- More parallelizable and requires significantly less training time
|
||||
|
||||
---
|
||||
|
||||
## Abstract: Performance Highlights
|
||||
|
||||
- Achieves 28.4 BLEU on WMT 2014 English-to-German
|
||||
- Improves over existing best results by over 2 BLEU
|
||||
- Establishes new state-of-the-art 41.8 BLEU on WMT 2014 English-to-French
|
||||
- Generalizes well to other tasks like English constituency parsing
|
||||
|
||||
---
|
||||
|
||||
## Background: Limitations of RNNs
|
||||
|
||||
### Recurrent Neural Networks (RNNs/LSTMs/GRUs)
|
||||
- Inherently sequential computation
|
||||
- Cannot parallelize within training examples
|
||||
- Difficult to learn long-range dependencies
|
||||
- Memory constraints limit batching for long sequences
|
||||
|
||||
---
|
||||
|
||||
## Background: Limitations of Convolutional Approaches
|
||||
|
||||
### CNN-based Models (ByteNet, ConvS2S)
|
||||
- Use convolutions for parallelization
|
||||
- Number of operations grows with distance between positions
|
||||
- Linear growth for ConvS2S
|
||||
- Logarithmic growth for ByteNet
|
||||
- Longer path lengths between distant positions
|
||||
|
||||
---
|
||||
|
||||
## Key Insight: Attention is Sufficient
|
||||
|
||||
Attention mechanisms allow modeling dependencies without regard to distance, but were previously used with RNNs.
|
||||
|
||||
**Transformer**: First transduction model relying entirely on self-attention to compute representations without:
|
||||
- Sequence-aligned RNNs
|
||||
- Convolutions
|
||||
|
||||
---
|
||||
|
||||
## Transformer Architecture Overview
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-80 mx-auto" />
|
||||
|
||||
*Encoder-decoder structure with stacked self-attention and feed-forward layers*
|
||||
|
||||
---
|
||||
|
||||
## Encoder Structure
|
||||
|
||||
- Stack of **6 identical layers**
|
||||
- Residual connections around each sub-layer
|
||||
- Layer normalization after each sub-layer
|
||||
- All sub-layers produce outputs of dimension `d_model = 512`
|
||||
|
||||
---
|
||||
|
||||
## Encoder: Sub-layer Details
|
||||
|
||||
Each encoder layer contains two sub-layers:
|
||||
|
||||
1. **Multi-head self-attention** mechanism
|
||||
- All positions attend to all positions in previous layer
|
||||
- Enables modeling of dependencies throughout sequence
|
||||
|
||||
2. **Position-wise fully connected feed-forward network**
|
||||
- Applied to each position separately and identically
|
||||
- Two linear transformations with ReLU activation
|
||||
|
||||
---
|
||||
|
||||
## Decoder Structure
|
||||
|
||||
- Stack of **6 identical layers**
|
||||
- Residual connections and layer normalization
|
||||
- Output embeddings offset by one position (auto-regressive property)
|
||||
|
||||
---
|
||||
|
||||
## Decoder: Sub-layer Details
|
||||
|
||||
Each decoder layer contains three sub-layers:
|
||||
|
||||
1. **Masked multi-head self-attention**
|
||||
- Prevents positions from attending to subsequent positions
|
||||
|
||||
2. **Multi-head attention over encoder output**
|
||||
- Queries from decoder, keys/values from encoder
|
||||
|
||||
3. **Position-wise fully connected feed-forward network**
|
||||
- Same structure as encoder's feed-forward network
|
||||
|
||||
---
|
||||
|
||||
## Attention Mechanism
|
||||
|
||||
### Scaled Dot-Product Attention
|
||||
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
|
||||
|
||||
- $Q$ (queries), $K$ (keys), $V$ (values) are matrices
|
||||
- Scaling by $\frac{1}{\sqrt{d_k}}$ prevents gradients from becoming too small
|
||||
- Faster and more space-efficient than additive attention
|
||||
|
||||
---
|
||||
|
||||
## Multi-Head Attention
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-60 mx-auto" />
|
||||
|
||||
- Projects queries, keys, values $h$ times with different learned projections
|
||||
- Performs attention in parallel on projected versions
|
||||
- Concatenates results and projects again
|
||||
|
||||
$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O$$
|
||||
|
||||
---
|
||||
|
||||
## Attention Application: Encoder-Decoder
|
||||
|
||||
**Encoder-decoder attention**:
|
||||
- Queries come from previous decoder layer
|
||||
- Memory keys and values come from encoder output
|
||||
- Allows every position in decoder to attend over all positions in input sequence
|
||||
- Mimics typical encoder-decoder attention mechanisms
|
||||
|
||||
---
|
||||
|
||||
## Attention Application: Encoder Self-Attention
|
||||
|
||||
**Encoder self-attention**:
|
||||
- Keys, values and queries all come from previous encoder layer
|
||||
- Each position attends to all positions in previous encoder layer
|
||||
- Enables modeling of relationships between all words in input sequence
|
||||
- No regard to distance between positions
|
||||
|
||||
---
|
||||
|
||||
## Attention Application: Decoder Self-Attention
|
||||
|
||||
**Decoder self-attention**:
|
||||
- Keys, values and queries come from previous decoder layer
|
||||
- Each position attends to all positions up to and including itself
|
||||
- Masking prevents attending to subsequent positions
|
||||
- Preserves auto-regressive property (predictions depend only on known outputs)
|
||||
|
||||
---
|
||||
|
||||
## Positional Encoding
|
||||
|
||||
Since model has no recurrence/convolution, we inject positional information:
|
||||
|
||||
$$\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
|
||||
$$\text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
|
||||
|
||||
- Same dimension as embeddings ($d_{\text{model}}$)
|
||||
- Allows model to learn relative position information
|
||||
- Performed nearly as well as learned positional embeddings
|
||||
|
||||
---
|
||||
|
||||
## Why Self-Attention?
|
||||
|
||||
| Layer Type | Complexity | Sequential Operations | Max Path Length |
|
||||
|------------|------------|-----------------------|-----------------|
|
||||
| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
|
||||
| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
|
||||
| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
|
||||
|
||||
---
|
||||
|
||||
## Training: Data & Batching
|
||||
|
||||
- WMT 2014 English-German (4.5M sentence pairs)
|
||||
- WMT 2014 English-French (36M sentence pairs)
|
||||
- Byte-pair encoding (37K shared vocab for EN-DE)
|
||||
- Batches with ~25000 source and target tokens
|
||||
|
||||
---
|
||||
|
||||
## Training: Hardware & Schedule
|
||||
|
||||
- 8 NVIDIA P100 GPUs
|
||||
- Base model: 100,000 steps (12 hours)
|
||||
- Big model: 300,000 steps (3.5 days)
|
||||
- Adam optimizer with scheduled learning rate
|
||||
|
||||
---
|
||||
|
||||
## Machine Translation Results
|
||||
|
||||
| Model | EN-DE BLEU | EN-FR BLEU | Training Cost (FLOPs) |
|
||||
|-------|------------|------------|-----------------------|
|
||||
| GNMT + RL Ensemble | 26.30 | 41.16 | $1.8 \cdot 10^{20}$ |
|
||||
| ConvS2S Ensemble | 26.36 | 41.29 | $7.7 \cdot 10^{19}$ |
|
||||
| **Transformer (big)** | **28.4** | **41.8** | **$2.3 \cdot 10^{19}$** |
|
||||
|
||||
---
|
||||
|
||||
## Generalization to Constituency Parsing
|
||||
|
||||
| Parser | Training | WSJ 23 F1 |
|
||||
|--------|----------|-----------|
|
||||
| Previous state-of-the-art | WSJ only | 91.7 |
|
||||
| **Transformer (4 layers)** | **WSJ only** | **91.3** |
|
||||
| Previous state-of-the-art | Semi-supervised | 92.1 |
|
||||
| **Transformer (4 layers)** | **Semi-supervised** | **92.7** |
|
||||
|
||||
---
|
||||
|
||||
## Attention Visualization: Long-Distance Dependencies
|
||||
|
||||
<img src="/paper_figure_3_long_distance.png" class="h-full w-full object-contain" />
|
||||
|
||||
---
|
||||
|
||||
## Attention Visualization: Anaphora Resolution
|
||||
|
||||
<img src="/paper_figure_4_anaphora.png" class="h-full w-full object-contain" />
|
||||
|
||||
---
|
||||
|
||||
## Limitations
|
||||
|
||||
- Computational complexity grows quadratically with sequence length
|
||||
- Less effective for very long sequences
|
||||
- Still requires sequential generation in decoder
|
||||
- Limited ability to model hierarchical structure
|
||||
|
||||
---
|
||||
|
||||
## Key Contributions
|
||||
|
||||
- Introduced Transformer architecture based solely on attention
|
||||
- Achieved new state-of-the-art results in machine translation
|
||||
- Demonstrated improved parallelization and reduced training time
|
||||
- Showed generalization to other tasks like constituency parsing
|
||||
|
||||
---
|
||||
|
||||
## Future Work
|
||||
|
||||
- Apply to other modalities (images, audio, video)
|
||||
- Investigate local, restricted attention for large inputs
|
||||
- Make generation less sequential
|
||||
- Explore interpretability of attention mechanisms
|
||||
|
||||
---
|
||||
|
||||
## Thank You
|
||||
|
||||
Code available at: https://github.com/tensorflow/tensor2tensor
|
||||
|
||||
arXiv:1706.03762v7 [cs.CL]
|
||||
+1367
File diff suppressed because one or more lines are too long
BIN
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 100 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 107 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 86 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 110 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 110 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 123 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 119 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 150 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 97 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 63 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 119 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 117 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 82 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 113 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 68 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 126 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 60 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 57 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 164 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 197 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 125 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 70 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 49 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 89 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 83 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 124 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 75 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 79 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 92 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 69 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 57 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 59 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 66 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 74 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 192 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 241 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 54 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 65 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 36 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 73 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 76 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 72 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 116 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 55 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 93 KiB |
+1111
File diff suppressed because it is too large
Load Diff
+62
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"paper": {
|
||||
"title": "Attention Is All You Need",
|
||||
"authors": "Ashish Vaswani et al.",
|
||||
"arxiv_id": "1706.03762",
|
||||
"pdf_url": "https://arxiv.org/pdf/1706.03762",
|
||||
"pdf_sha256": "bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697",
|
||||
"observed_pdf_sha256": "bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697"
|
||||
},
|
||||
"paper_text": {
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/source/paper_text.md",
|
||||
"sha256": "4da5f2a1da38bad8149832b5ada3c94f0619188db8fed5cdca8ff63405587eb8",
|
||||
"characters": 39819
|
||||
},
|
||||
"visuals": [
|
||||
{
|
||||
"filename": "paper_figure_1_transformer.png",
|
||||
"pdf_page": 3,
|
||||
"source_label": "Figure 1",
|
||||
"caption": "The Transformer model architecture.",
|
||||
"rect": [
|
||||
92,
|
||||
28,
|
||||
520,
|
||||
535
|
||||
],
|
||||
"sha256": "48f09e2e2426e8ca76c1e438600438dca2f9e61360e92ae3edbb538a2cc611d2",
|
||||
"bytes": 150087,
|
||||
"public_copy_sha256": "48f09e2e2426e8ca76c1e438600438dca2f9e61360e92ae3edbb538a2cc611d2"
|
||||
},
|
||||
{
|
||||
"filename": "paper_figure_3_long_distance.png",
|
||||
"pdf_page": 13,
|
||||
"source_label": "Figure 3",
|
||||
"caption": "Encoder self-attention following long-distance dependencies.",
|
||||
"rect": [
|
||||
92,
|
||||
55,
|
||||
525,
|
||||
455
|
||||
],
|
||||
"sha256": "ea456861962564d2ef5a158e4205cfea29b537f4a611df989db5d65471f23d78",
|
||||
"bytes": 96204,
|
||||
"public_copy_sha256": "ea456861962564d2ef5a158e4205cfea29b537f4a611df989db5d65471f23d78"
|
||||
},
|
||||
{
|
||||
"filename": "paper_figure_4_anaphora.png",
|
||||
"pdf_page": 14,
|
||||
"source_label": "Figure 4",
|
||||
"caption": "Attention heads involved in anaphora resolution.",
|
||||
"rect": [
|
||||
92,
|
||||
135,
|
||||
525,
|
||||
665
|
||||
],
|
||||
"sha256": "b1f0c6b0e186acb50cae69c304a9a2a0e1f09fd15f1beb0bba03194528163d7e",
|
||||
"bytes": 264855,
|
||||
"public_copy_sha256": "b1f0c6b0e186acb50cae69c304a9a2a0e1f09fd15f1beb0bba03194528163d7e"
|
||||
}
|
||||
]
|
||||
}
|
||||
BIN
Binary file not shown.
|
After Width: | Height: | Size: 147 KiB |
BIN
Binary file not shown.
|
After Width: | Height: | Size: 94 KiB |
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user