ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）

chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/dual_round1_review.json

@@ -0,0 +1,90 @@
+{
+  "overall_score": 65,
+  "pass": false,
+  "issues": [
+    {
+      "page": 2,
+      "issue_type": "overcrowded",
+      "severity": "medium",
+      "suggestion": "将6个要点拆分为2页，每页不超过3-4个要点，或精简合并相似内容"
+    },
+    {
+      "page": 3,
+      "issue_type": "overcrowded",
+      "severity": "medium",
+      "suggestion": "将RNN和卷积方法分两页展示，或每个部分保留2个核心要点"
+    },
+    {
+      "page": 5,
+      "issue_type": "image_size",
+      "severity": "high",
+      "suggestion": "放大架构图至至少占页面60%空间，确保图中文字和模块清晰可见"
+    },
+    {
+      "page": 5,
+      "issue_type": "readability",
+      "severity": "medium",
+      "suggestion": "删除图片下方的小字体说明文字，或单独用一页解释架构细节"
+    },
+    {
+      "page": 7,
+      "issue_type": "overcrowded",
+      "severity": "medium",
+      "suggestion": "将3个子要点拆分为独立要点，或移至下一页单独讲解decoder子层结构"
+    },
+    {
+      "page": 9,
+      "issue_type": "image_size",
+      "severity": "high",
+      "suggestion": "放大多头注意力机制图，确保每个头的结构和连接关系清晰可辨"
+    },
+    {
+      "page": 9,
+      "issue_type": "readability",
+      "severity": "medium",
+      "suggestion": "删除图片下方的小字体说明文字，公式保持现有大小即可"
+    },
+    {
+      "page": 13,
+      "issue_type": "overcrowded",
+      "severity": "medium",
+      "suggestion": "将Data & Batching和Hardware & Schedule分两页展示，或各保留2个最关键要点"
+    },
+    {
+      "page": 15,
+      "issue_type": "overcrowded",
+      "severity": "medium",
+      "suggestion": "表格保留核心对比数据，4个要点精简为2-3个核心结论"
+    },
+    {
+      "page": 18,
+      "issue_type": "image_size",
+      "severity": "high",
+      "suggestion": "单独一页放大展示注意力可视化图，确保能看清词语间的注意力连接关系"
+    },
+    {
+      "page": 18,
+      "issue_type": "readability",
+      "severity": "medium",
+      "suggestion": "删除图片下方的小字体说明文字，用简洁标题概括可视化内容"
+    },
+    {
+      "page": 19,
+      "issue_type": "image_size",
+      "severity": "high",
+      "suggestion": "单独一页放大展示指代消解可视化图，确保紫色连接线和词语清晰可见"
+    },
+    {
+      "page": 19,
+      "issue_type": "readability",
+      "severity": "medium",
+      "suggestion": "删除图片下方的小字体说明文字，直接在标题中说明可视化内容"
+    },
+    {
+      "page": 21,
+      "issue_type": "overcrowded",
+      "severity": "medium",
+      "suggestion": "将Key Contributions和Future Directions分两页展示，每部分保留3-4个要点"
+    }
+  ]
+}

chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/dual_round1_slides.md

@@ -0,0 +1,261 @@
+---
+theme: default
+---
+
+# Attention Is All You Need
+## A Revolutionary Architecture for Sequence Transduction
+
+Ashish Vaswani et al.  
+NIPS 2017
+
+---
+
+## Abstract
+
+- Proposes **Transformer** - first model based solely on attention mechanisms
+- Dispenses with recurrence and convolutions entirely
+- Superior quality while being more parallelizable and requiring less training time
+- Achieves 28.4 BLEU on WMT 2014 English-to-German (↑2 BLEU over previous best)
+- Achieves 41.8 BLEU on WMT 2014 English-to-French (new state-of-the-art)
+- Generalizes well to other tasks like English constituency parsing
+
+---
+
+## Background: The Problem with Existing Approaches
+
+### Recurrent Neural Networks (RNNs/LSTMs/GRUs)
+- Inherently sequential computation
+- Cannot parallelize within training examples
+- Difficult to learn long-range dependencies
+
+### Convolutional Approaches
+- ByteNet, ConvS2S use CNNs for parallelization
+- Number of operations grows with distance between positions
+- Linear (ConvS2S) or logarithmic (ByteNet) path lengths
+
+---
+
+## Key Insight: Attention is Sufficient
+
+Attention mechanisms allow modeling dependencies without regard to distance, but were previously used with RNNs.
+
+**Transformer**: First transduction model relying entirely on self-attention to compute representations without:
+- Sequence-aligned RNNs
+- Convolutions
+
+---
+
+## Transformer Architecture Overview
+
+<img src="/paper_figure_1_transformer.png" class="h-80 mx-auto" />
+
+*Encoder-decoder structure with stacked self-attention and feed-forward layers*
+
+---
+
+## Encoder Structure
+
+- Stack of **6 identical layers**
+- Each layer has two sub-layers:
+  1. **Multi-head self-attention** mechanism
+  2. **Position-wise fully connected feed-forward network**
+- Residual connections around each sub-layer
+- Layer normalization after each sub-layer
+- All sub-layers produce outputs of dimension `d_model = 512`
+
+---
+
+## Decoder Structure
+
+- Stack of **6 identical layers**
+- Three sub-layers per layer:
+  1. Masked multi-head self-attention (prevents leftward information flow)
+  2. Multi-head attention over encoder output
+  3. Position-wise fully connected feed-forward network
+- Residual connections and layer normalization
+- Output embeddings offset by one position (auto-regressive property)
+
+---
+
+## Attention Mechanism
+
+### Scaled Dot-Product Attention
+$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
+
+- $Q$ (queries), $K$ (keys), $V$ (values) are matrices
+- Scaling by $\frac{1}{\sqrt{d_k}}$ prevents gradients from becoming too small
+- Faster and more space-efficient than additive attention
+
+---
+
+## Multi-Head Attention
+
+<img src="/paper_figure_1_transformer.png" class="h-40 mx-auto" />
+
+- Projects queries, keys, values $h$ times with different learned projections
+- Performs attention in parallel on projected versions
+- Concatenates results and projects again
+
+$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O$$
+where $\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$
+
+---
+
+## Three Applications of Attention
+
+1. **Encoder-decoder attention**: Queries from decoder, keys/values from encoder
+2. **Encoder self-attention**: All keys, values, queries from previous encoder layer
+3. **Decoder self-attention**: All positions in decoder up to current position (masked)
+
+---
+
+## Positional Encoding
+
+Since model has no recurrence/convolution, we inject positional information:
+
+$$\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
+$$\text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
+
+- Same dimension as embeddings ($d_{\text{model}}$)
+- Allows model to learn relative position information
+- Performed nearly as well as learned positional embeddings
+
+---
+
+## Why Self-Attention?
+
+| Layer Type | Complexity | Sequential Operations | Max Path Length |
+|------------|------------|-----------------------|-----------------|
+| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
+| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
+| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
+
+- Constant path length between any positions
+- More parallelizable than RNNs
+- Better computational efficiency for typical sentence lengths
+
+---
+
+## Training Details
+
+### Data & Batching
+- WMT 2014 English-German (4.5M sentence pairs)
+- WMT 2014 English-French (36M sentence pairs)
+- Byte-pair encoding (37K shared vocab for EN-DE)
+- Batches with ~25000 source and target tokens
+
+### Hardware & Schedule
+- 8 NVIDIA P100 GPUs
+- Base model: 100,000 steps (12 hours)
+- Big model: 300,000 steps (3.5 days)
+
+---
+
+## Training Details (Cont.)
+
+### Optimizer
+- Adam with $\beta_1 = 0.9$, $\beta_2 = 0.98$, $\epsilon = 10^{-9}$
+- Learning rate schedule:
+  $$\text{lrate} = d_{\text{model}}^{-0.5} \cdot \min(\text{step\_num}^{-0.5}, \text{step\_num} \cdot \text{warmup\_steps}^{-1.5})$$
+- Warmup steps = 4000
+
+### Regularization
+- Residual dropout (P_drop = 0.1)
+- Label smoothing ($\epsilon_{ls} = 0.1$)
+
+---
+
+## Machine Translation Results
+
+| Model | EN-DE BLEU | EN-FR BLEU | Training Cost (FLOPs) |
+|-------|------------|------------|-----------------------|
+| GNMT + RL Ensemble | 26.30 | 41.16 | $1.8 \cdot 10^{20}$ |
+| ConvS2S Ensemble | 26.36 | 41.29 | $7.7 \cdot 10^{19}$ |
+| **Transformer (big)** | **28.4** | **41.8** | **$2.3 \cdot 10^{19}$** |
+
+- Transformer outperforms all previous state-of-the-art models
+- Achieves better results with significantly lower training cost
+- 28.4 BLEU on EN-DE (↑2 BLEU over previous best)
+- 41.8 BLEU on EN-FR (new state-of-the-art)
+
+---
+
+## Model Variations Analysis
+
+| Variation | Dev PPL | Dev BLEU |
+|-----------|---------|----------|
+| Base model | 4.92 | 25.8 |
+| Single attention head | 5.29 | 24.9 |
+| No dropout | 5.77 | 24.6 |
+| Learned positional embeddings | 4.92 | 25.7 |
+| Big model | 4.33 | 26.4 |
+
+- Multiple attention heads improve performance
+- Dropout is crucial for avoiding overfitting
+- Sinusoidal and learned positional encodings perform similarly
+- Larger models (more dimensions, more heads) improve performance
+
+---
+
+## Generalization to Other Tasks: English Constituency Parsing
+
+| Parser | Training | WSJ 23 F1 |
+|--------|----------|-----------|
+| Petrov et al. (2006) | WSJ only | 90.4 |
+| Dyer et al. (2016) | WSJ only | 91.7 |
+| **Transformer (4 layers)** | **WSJ only** | **91.3** |
+| Vinyals & Kaiser et al. | Semi-supervised | 92.1 |
+| **Transformer (4 layers)** | **Semi-supervised** | **92.7** |
+
+- Transformer performs well despite no task-specific tuning
+- Outperforms previous models in semi-supervised setting
+- Shows generalization ability beyond machine translation
+
+---
+
+## Attention Visualization: Long-Distance Dependencies
+
+<img src="/paper_figure_3_long_distance.png" class="h-70 mx-auto" />
+
+*Encoder self-attention in layer 5 showing attention to distant dependency of the verb "making"*
+
+---
+
+## Attention Visualization: Anaphora Resolution
+
+<img src="/paper_figure_4_anaphora.png" class="h-70 mx-auto" />
+
+*Attention heads involved in resolving "its" reference to "The Law"*
+
+---
+
+## Limitations
+
+- Computational complexity grows quadratically with sequence length
+- Less effective for very long sequences (e.g., books, articles)
+- Still requires sequential generation in decoder
+- Limited ability to model hierarchical structure compared to some syntactic models
+
+---
+
+## Conclusion and Future Work
+
+### Key Contributions
+- Introduced Transformer architecture based solely on attention
+- Achieved new state-of-the-art results in machine translation
+- Demonstrated improved parallelization and reduced training time
+- Showed generalization to other tasks like constituency parsing
+
+### Future Directions
+- Apply to other modalities (images, audio, video)
+- Investigate local, restricted attention for large inputs
+- Make generation less sequential
+- Explore interpretability of attention mechanisms
+
+---
+
+## Thank You
+
+Code available at: https://github.com/tensorflow/tensor2tensor
+
+arXiv:1706.03762v7 [cs.CL]

chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/dual_round2_review.json

@@ -0,0 +1,42 @@
+{
+  "overall_score": 65,
+  "pass": false,
+  "issues": [
+    {
+      "page": 7,
+      "issue_type": "image_size",
+      "severity": "medium",
+      "suggestion": "缩小架构图尺寸，确保下方说明文字有足够空间，避免图片挤压文字区域"
+    },
+    {
+      "page": 8,
+      "issue_type": "overcrowded",
+      "severity": "medium",
+      "suggestion": "将5个要点拆分为两页，建议将\"每个层包含两个子层\"及其子要点单独成页"
+    },
+    {
+      "page": 12,
+      "issue_type": "image_size",
+      "severity": "medium",
+      "suggestion": "调整多头注意力架构图大小，确保图片下方的文字说明完整可见且易于阅读"
+    },
+    {
+      "page": 13,
+      "issue_type": "overcrowded",
+      "severity": "low",
+      "suggestion": "将三个注意力应用拆分为单独页面，或简化每个应用的子要点描述"
+    },
+    {
+      "page": 21,
+      "issue_type": "image_size",
+      "severity": "high",
+      "suggestion": "大幅放大注意力可视化图，确保图中文字和颜色编码清晰可辨，必要时单独成页展示"
+    },
+    {
+      "page": 22,
+      "issue_type": "image_size",
+      "severity": "high",
+      "suggestion": "放大指代消解可视化图，增加图片分辨率，确保线条和文字清晰可读"
+    }
+  ]
+}

chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/dual_round2_slides.md

@@ -0,0 +1,256 @@
+---
+theme: default
+---
+
+# Attention Is All You Need
+## A Revolutionary Architecture for Sequence Transduction
+
+Ashish Vaswani et al.  
+NIPS 2017
+
+---
+
+## Abstract: Key Innovation
+
+- Proposes **Transformer** - first model based solely on attention mechanisms
+- Dispenses with recurrence and convolutions entirely
+- More parallelizable and requires significantly less training time
+
+---
+
+## Abstract: Performance Highlights
+
+- Achieves 28.4 BLEU on WMT 2014 English-to-German
+  - Improves over existing best results by over 2 BLEU
+- Establishes new state-of-the-art 41.8 BLEU on WMT 2014 English-to-French
+- Generalizes well to other tasks like English constituency parsing
+
+---
+
+## Background: Limitations of RNNs
+
+### Recurrent Neural Networks (RNNs/LSTMs/GRUs)
+- Inherently sequential computation
+- Cannot parallelize within training examples
+- Difficult to learn long-range dependencies
+- Memory constraints limit batching for long sequences
+
+---
+
+## Background: Limitations of Convolutional Approaches
+
+### CNN-based Models (ByteNet, ConvS2S)
+- Use convolutions for parallelization
+- Number of operations grows with distance between positions
+  - Linear growth for ConvS2S
+  - Logarithmic growth for ByteNet
+- Longer path lengths between distant positions
+
+---
+
+## Key Insight: Attention is Sufficient
+
+Attention mechanisms allow modeling dependencies without regard to distance, but were previously used with RNNs.
+
+**Transformer**: First transduction model relying entirely on self-attention to compute representations without:
+- Sequence-aligned RNNs
+- Convolutions
+
+---
+
+## Transformer Architecture Overview
+
+<img src="/paper_figure_1_transformer.png" class="h-96 mx-auto" />
+
+---
+
+## Encoder Structure
+
+- Stack of **6 identical layers**
+- Each layer contains two sub-layers:
+  1. Multi-head self-attention mechanism
+  2. Position-wise fully connected feed-forward network
+- Residual connections around each sub-layer
+- Layer normalization after each sub-layer
+
+---
+
+## Decoder Structure
+
+- Stack of **6 identical layers**
+- Three sub-layers per layer:
+  1. Masked multi-head self-attention
+  2. Multi-head attention over encoder output
+  3. Position-wise fully connected feed-forward network
+- Residual connections and layer normalization
+
+---
+
+## Decoder: Masking Mechanism
+
+- Prevents positions from attending to subsequent positions
+- Ensures predictions for position i depend only on:
+  - Known outputs at positions < i
+  - Output embeddings offset by one position
+- Preserves auto-regressive property
+
+---
+
+## Attention Mechanism
+
+### Scaled Dot-Product Attention
+$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
+
+- $Q$ (queries), $K$ (keys), $V$ (values) are matrices
+- Scaling by $\frac{1}{\sqrt{d_k}}$ prevents gradients from becoming too small
+- Faster and more space-efficient than additive attention
+
+---
+
+## Multi-Head Attention
+
+<img src="/paper_figure_1_transformer.png" class="h-80 mx-auto" />
+
+- Projects queries, keys, values $h$ times with different learned projections
+- Performs attention in parallel on projected versions
+- Concatenates results and projects again
+
+---
+
+## Three Applications of Attention
+
+1. **Encoder-decoder attention**: 
+   - Queries from decoder
+   - Keys/values from encoder output
+
+2. **Encoder self-attention**:
+   - All keys, values, queries from previous encoder layer
+   - Each position attends to all positions
+
+3. **Decoder self-attention**:
+   - All positions in decoder up to current position
+   - Masked to prevent future information flow
+
+---
+
+## Positional Encoding
+
+Since model has no recurrence/convolution, we inject positional information:
+
+$$\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
+$$\text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
+
+- Same dimension as embeddings ($d_{\text{model}}$)
+- Allows model to learn relative position information
+- Performed nearly as well as learned positional embeddings
+
+---
+
+## Why Self-Attention?
+
+| Layer Type | Complexity | Sequential Operations | Max Path Length |
+|------------|------------|-----------------------|-----------------|
+| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
+| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
+| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
+
+---
+
+## Training: Data & Batching
+
+- WMT 2014 English-German (4.5M sentence pairs)
+- WMT 2014 English-French (36M sentence pairs)
+- Byte-pair encoding (37K shared vocab for EN-DE)
+- Batches with ~25000 source and target tokens
+
+---
+
+## Training: Hardware & Schedule
+
+- 8 NVIDIA P100 GPUs
+- Base model: 100,000 steps (12 hours)
+- Big model: 300,000 steps (3.5 days)
+- Adam optimizer with scheduled learning rate
+
+---
+
+## Machine Translation Results
+
+| Model | EN-DE BLEU | EN-FR BLEU | Training Cost (FLOPs) |
+|-------|------------|------------|-----------------------|
+| GNMT + RL Ensemble | 26.30 | 41.16 | $1.8 \cdot 10^{20}$ |
+| ConvS2S Ensemble | 26.36 | 41.29 | $7.7 \cdot 10^{19}$ |
+| **Transformer (big)** | **28.4** | **41.8** | **$2.3 \cdot 10^{19}$** |
+
+---
+
+## Model Variations Analysis
+
+| Variation | Dev PPL | Dev BLEU |
+|-----------|---------|----------|
+| Base model | 4.92 | 25.8 |
+| Single attention head | 5.29 | 24.9 |
+| No dropout | 5.77 | 24.6 |
+| Learned positional embeddings | 4.92 | 25.7 |
+
+---
+
+## Generalization to Constituency Parsing
+
+| Parser | Training | WSJ 23 F1 |
+|--------|----------|-----------|
+| Previous state-of-the-art | WSJ only | 91.7 |
+| **Transformer (4 layers)** | **WSJ only** | **91.3** |
+| Previous state-of-the-art | Semi-supervised | 92.1 |
+| **Transformer (4 layers)** | **Semi-supervised** | **92.7** |
+
+---
+
+## Attention Visualization: Long-Distance Dependencies
+
+<img src="/paper_figure_3_long_distance.png" class="h-96 mx-auto" />
+
+*Encoder self-attention showing long-distance dependency for "making"*
+
+---
+
+## Attention Visualization: Anaphora Resolution
+
+<img src="/paper_figure_4_anaphora.png" class="h-96 mx-auto" />
+
+*Attention heads resolving "its" reference to "The Law"*
+
+---
+
+## Limitations
+
+- Computational complexity grows quadratically with sequence length
+- Less effective for very long sequences
+- Still requires sequential generation in decoder
+- Limited ability to model hierarchical structure
+
+---
+
+## Key Contributions
+
+- Introduced Transformer architecture based solely on attention
+- Achieved new state-of-the-art results in machine translation
+- Demonstrated improved parallelization and reduced training time
+- Showed generalization to other tasks like constituency parsing
+
+---
+
+## Future Work
+
+- Apply to other modalities (images, audio, video)
+- Investigate local, restricted attention for large inputs
+- Make generation less sequential
+- Explore interpretability of attention mechanisms
+
+---
+
+## Thank You
+
+Code available at: https://github.com/tensorflow/tensor2tensor
+
+arXiv:1706.03762v7 [cs.CL]

chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/dual_round3_review.json

@@ -0,0 +1,18 @@
+{
+  "overall_score": 85,
+  "pass": false,
+  "issues": [
+    {
+      "page": 23,
+      "issue_type": "readability",
+      "severity": "medium",
+      "suggestion": "原图彩色注意力可视化转为黑白后对比度差，不同注意力头难以区分。建议添加灰度区分或图案填充来标识不同注意力头，或增加图例说明关键颜色对应的含义。"
+    },
+    {
+      "page": 24,
+      "issue_type": "readability",
+      "severity": "medium",
+      "suggestion": "原图彩色注意力可视化转为黑白后线条区分度低。建议使用不同线型（实线/虚线/点线）或线宽来区分不同注意力头，确保黑白打印下仍能清晰识别关键依赖关系。"
+    }
+  ]
+}

chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/dual_round3_slides.md

@@ -0,0 +1,273 @@
+---
+theme: default
+---
+
+# Attention Is All You Need
+## A Revolutionary Architecture for Sequence Transduction
+
+Ashish Vaswani et al.  
+NIPS 2017
+
+---
+
+## Abstract: Key Innovation
+
+- Proposes **Transformer** - first model based solely on attention mechanisms
+- Dispenses with recurrence and convolutions entirely
+- More parallelizable and requires significantly less training time
+
+---
+
+## Abstract: Performance Highlights
+
+- Achieves 28.4 BLEU on WMT 2014 English-to-German
+  - Improves over existing best results by over 2 BLEU
+- Establishes new state-of-the-art 41.8 BLEU on WMT 2014 English-to-French
+- Generalizes well to other tasks like English constituency parsing
+
+---
+
+## Background: Limitations of RNNs
+
+### Recurrent Neural Networks (RNNs/LSTMs/GRUs)
+- Inherently sequential computation
+- Cannot parallelize within training examples
+- Difficult to learn long-range dependencies
+- Memory constraints limit batching for long sequences
+
+---
+
+## Background: Limitations of Convolutional Approaches
+
+### CNN-based Models (ByteNet, ConvS2S)
+- Use convolutions for parallelization
+- Number of operations grows with distance between positions
+  - Linear growth for ConvS2S
+  - Logarithmic growth for ByteNet
+- Longer path lengths between distant positions
+
+---
+
+## Key Insight: Attention is Sufficient
+
+Attention mechanisms allow modeling dependencies without regard to distance, but were previously used with RNNs.
+
+**Transformer**: First transduction model relying entirely on self-attention to compute representations without:
+- Sequence-aligned RNNs
+- Convolutions
+
+---
+
+## Transformer Architecture Overview
+
+<img src="/paper_figure_1_transformer.png" class="h-80 mx-auto" />
+
+*Encoder-decoder structure with stacked self-attention and feed-forward layers*
+
+---
+
+## Encoder Structure
+
+- Stack of **6 identical layers**
+- Residual connections around each sub-layer
+- Layer normalization after each sub-layer
+- All sub-layers produce outputs of dimension `d_model = 512`
+
+---
+
+## Encoder: Sub-layer Details
+
+Each encoder layer contains two sub-layers:
+
+1. **Multi-head self-attention** mechanism
+   - All positions attend to all positions in previous layer
+   - Enables modeling of dependencies throughout sequence
+
+2. **Position-wise fully connected feed-forward network**
+   - Applied to each position separately and identically
+   - Two linear transformations with ReLU activation
+
+---
+
+## Decoder Structure
+
+- Stack of **6 identical layers**
+- Residual connections and layer normalization
+- Output embeddings offset by one position (auto-regressive property)
+
+---
+
+## Decoder: Sub-layer Details
+
+Each decoder layer contains three sub-layers:
+
+1. **Masked multi-head self-attention**
+   - Prevents positions from attending to subsequent positions
+
+2. **Multi-head attention over encoder output**
+   - Queries from decoder, keys/values from encoder
+
+3. **Position-wise fully connected feed-forward network**
+   - Same structure as encoder's feed-forward network
+
+---
+
+## Attention Mechanism
+
+### Scaled Dot-Product Attention
+$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
+
+- $Q$ (queries), $K$ (keys), $V$ (values) are matrices
+- Scaling by $\frac{1}{\sqrt{d_k}}$ prevents gradients from becoming too small
+- Faster and more space-efficient than additive attention
+
+---
+
+## Multi-Head Attention
+
+<img src="/paper_figure_1_transformer.png" class="h-60 mx-auto" />
+
+- Projects queries, keys, values $h$ times with different learned projections
+- Performs attention in parallel on projected versions
+- Concatenates results and projects again
+
+$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O$$
+
+---
+
+## Attention Application: Encoder-Decoder
+
+**Encoder-decoder attention**:
+- Queries come from previous decoder layer
+- Memory keys and values come from encoder output
+- Allows every position in decoder to attend over all positions in input sequence
+- Mimics typical encoder-decoder attention mechanisms
+
+---
+
+## Attention Application: Encoder Self-Attention
+
+**Encoder self-attention**:
+- Keys, values and queries all come from previous encoder layer
+- Each position attends to all positions in previous encoder layer
+- Enables modeling of relationships between all words in input sequence
+- No regard to distance between positions
+
+---
+
+## Attention Application: Decoder Self-Attention
+
+**Decoder self-attention**:
+- Keys, values and queries come from previous decoder layer
+- Each position attends to all positions up to and including itself
+- Masking prevents attending to subsequent positions
+- Preserves auto-regressive property (predictions depend only on known outputs)
+
+---
+
+## Positional Encoding
+
+Since model has no recurrence/convolution, we inject positional information:
+
+$$\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
+$$\text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)$$
+
+- Same dimension as embeddings ($d_{\text{model}}$)
+- Allows model to learn relative position information
+- Performed nearly as well as learned positional embeddings
+
+---
+
+## Why Self-Attention?
+
+| Layer Type | Complexity | Sequential Operations | Max Path Length |
+|------------|------------|-----------------------|-----------------|
+| Self-Attention | $O(n^2 \cdot d)$ | $O(1)$ | $O(1)$ |
+| Recurrent | $O(n \cdot d^2)$ | $O(n)$ | $O(n)$ |
+| Convolutional | $O(k \cdot n \cdot d^2)$ | $O(1)$ | $O(\log_k n)$ |
+
+---
+
+## Training: Data & Batching
+
+- WMT 2014 English-German (4.5M sentence pairs)
+- WMT 2014 English-French (36M sentence pairs)
+- Byte-pair encoding (37K shared vocab for EN-DE)
+- Batches with ~25000 source and target tokens
+
+---
+
+## Training: Hardware & Schedule
+
+- 8 NVIDIA P100 GPUs
+- Base model: 100,000 steps (12 hours)
+- Big model: 300,000 steps (3.5 days)
+- Adam optimizer with scheduled learning rate
+
+---
+
+## Machine Translation Results
+
+| Model | EN-DE BLEU | EN-FR BLEU | Training Cost (FLOPs) |
+|-------|------------|------------|-----------------------|
+| GNMT + RL Ensemble | 26.30 | 41.16 | $1.8 \cdot 10^{20}$ |
+| ConvS2S Ensemble | 26.36 | 41.29 | $7.7 \cdot 10^{19}$ |
+| **Transformer (big)** | **28.4** | **41.8** | **$2.3 \cdot 10^{19}$** |
+
+---
+
+## Generalization to Constituency Parsing
+
+| Parser | Training | WSJ 23 F1 |
+|--------|----------|-----------|
+| Previous state-of-the-art | WSJ only | 91.7 |
+| **Transformer (4 layers)** | **WSJ only** | **91.3** |
+| Previous state-of-the-art | Semi-supervised | 92.1 |
+| **Transformer (4 layers)** | **Semi-supervised** | **92.7** |
+
+---
+
+## Attention Visualization: Long-Distance Dependencies
+
+<img src="/paper_figure_3_long_distance.png" class="h-full w-full object-contain" />
+
+---
+
+## Attention Visualization: Anaphora Resolution
+
+<img src="/paper_figure_4_anaphora.png" class="h-full w-full object-contain" />
+
+---
+
+## Limitations
+
+- Computational complexity grows quadratically with sequence length
+- Less effective for very long sequences
+- Still requires sequential generation in decoder
+- Limited ability to model hierarchical structure
+
+---
+
+## Key Contributions
+
+- Introduced Transformer architecture based solely on attention
+- Achieved new state-of-the-art results in machine translation
+- Demonstrated improved parallelization and reduced training time
+- Showed generalization to other tasks like constituency parsing
+
+---
+
+## Future Work
+
+- Apply to other modalities (images, audio, video)
+- Investigate local, restricted attention for large inputs
+- Make generation less sequential
+- Explore interpretability of attention mechanisms
+
+---
+
+## Thank You
+
+Code available at: https://github.com/tensorflow/tensor2tensor
+
+arXiv:1706.03762v7 [cs.CL]

chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/source/source_visuals/manifest.json

@@ -0,0 +1,62 @@
+{
+  "paper": {
+    "title": "Attention Is All You Need",
+    "authors": "Ashish Vaswani et al.",
+    "arxiv_id": "1706.03762",
+    "pdf_url": "https://arxiv.org/pdf/1706.03762",
+    "pdf_sha256": "bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697",
+    "observed_pdf_sha256": "bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697"
+  },
+  "paper_text": {
+    "path": "/Users/boj/book/ai-agent-book/chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/source/paper_text.md",
+    "sha256": "4da5f2a1da38bad8149832b5ada3c94f0619188db8fed5cdca8ff63405587eb8",
+    "characters": 39819
+  },
+  "visuals": [
+    {
+      "filename": "paper_figure_1_transformer.png",
+      "pdf_page": 3,
+      "source_label": "Figure 1",
+      "caption": "The Transformer model architecture.",
+      "rect": [
+        92,
+        28,
+        520,
+        535
+      ],
+      "sha256": "48f09e2e2426e8ca76c1e438600438dca2f9e61360e92ae3edbb538a2cc611d2",
+      "bytes": 150087,
+      "public_copy_sha256": "48f09e2e2426e8ca76c1e438600438dca2f9e61360e92ae3edbb538a2cc611d2"
+    },
+    {
+      "filename": "paper_figure_3_long_distance.png",
+      "pdf_page": 13,
+      "source_label": "Figure 3",
+      "caption": "Encoder self-attention following long-distance dependencies.",
+      "rect": [
+        92,
+        55,
+        525,
+        455
+      ],
+      "sha256": "ea456861962564d2ef5a158e4205cfea29b537f4a611df989db5d65471f23d78",
+      "bytes": 96204,
+      "public_copy_sha256": "ea456861962564d2ef5a158e4205cfea29b537f4a611df989db5d65471f23d78"
+    },
+    {
+      "filename": "paper_figure_4_anaphora.png",
+      "pdf_page": 14,
+      "source_label": "Figure 4",
+      "caption": "Attention heads involved in anaphora resolution.",
+      "rect": [
+        92,
+        135,
+        525,
+        665
+      ],
+      "sha256": "b1f0c6b0e186acb50cae69c304a9a2a0e1f09fd15f1beb0bba03194528163d7e",
+      "bytes": 264855,
+      "public_copy_sha256": "b1f0c6b0e186acb50cae69c304a9a2a0e1f09fd15f1beb0bba03194528163d7e"
+    }
+  ]
+}