ai-agent-book 精选快照(<2MB 代码与文档,来自 github.com/bojieli/ai-agent-book)
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
@@ -0,0 +1,66 @@
|
||||
{
|
||||
"overall_score": 55,
|
||||
"pass": false,
|
||||
"issues": [
|
||||
{
|
||||
"page": 2,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将6个要点拆分为2页,例如将BLEU分数单独作为一页突出展示关键结果"
|
||||
},
|
||||
{
|
||||
"page": 3,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将传统方法的挑战拆分为单独页面,或减少子要点数量至2个以内"
|
||||
},
|
||||
{
|
||||
"page": 4,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将现有模型局限性和注意力机制分为2页,每部分保留3个核心要点"
|
||||
},
|
||||
{
|
||||
"page": 5,
|
||||
"issue_type": "image_size",
|
||||
"severity": "high",
|
||||
"suggestion": "增大架构图尺寸并移除图片下方的文字说明,或单独用一页展示清晰的架构图"
|
||||
},
|
||||
{
|
||||
"page": 5,
|
||||
"issue_type": "readability",
|
||||
"severity": "high",
|
||||
"suggestion": "确保架构图中的文字清晰可辨,必要时拆分图表或局部放大关键组件"
|
||||
},
|
||||
{
|
||||
"page": 5,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "删除图片下方的长段文字说明,仅保留底部的总结句"
|
||||
},
|
||||
{
|
||||
"page": 6,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将编码器架构的子层信息拆分为单独页面,或简化为要点列表"
|
||||
},
|
||||
{
|
||||
"page": 7,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将解码器的三个子层拆分为单独页面详细说明,主页面仅保留核心结构概述"
|
||||
},
|
||||
{
|
||||
"page": 8,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "low",
|
||||
"suggestion": "缩短定义部分的描述文字,或拆分为单独的'注意力定义'和'注意力类型'两页"
|
||||
},
|
||||
{
|
||||
"page": 10,
|
||||
"issue_type": "layout",
|
||||
"severity": "high",
|
||||
"suggestion": "补充Multi-Head Attention的核心要点(如并行注意力头、拼接输出等),避免空页"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
---
|
||||
theme: default
|
||||
title: "Attention Is All You Need"
|
||||
author: "Ashish Vaswani et al."
|
||||
---
|
||||
|
||||
# Attention Is All You Need
|
||||
|
||||
**Authors:** Ashish Vaswani et al.
|
||||
**Conference:** NIPS 2017
|
||||
|
||||
---
|
||||
|
||||
## Abstract
|
||||
|
||||
- Proposes **Transformer** - a new network architecture based solely on attention
|
||||
- Dispenses with recurrence and convolutions entirely
|
||||
- More parallelizable and requires significantly less training time
|
||||
- Achieves state-of-the-art results on machine translation tasks:
|
||||
- 28.4 BLEU on WMT 2014 English-to-German
|
||||
- 41.8 BLEU on WMT 2014 English-to-French
|
||||
- Generalizes well to other tasks like English constituency parsing
|
||||
|
||||
---
|
||||
|
||||
## Background: Sequence Modeling Challenges
|
||||
|
||||
### Traditional Approaches
|
||||
- **Recurrent models** (LSTM, GRU) process input sequentially
|
||||
- **Convolutional models** have limited receptive fields
|
||||
- Both struggle with:
|
||||
- Long-range dependencies
|
||||
- Parallelization
|
||||
- Computational efficiency for long sequences
|
||||
|
||||
---
|
||||
|
||||
## The Need for a New Architecture
|
||||
|
||||
### Key Limitations of Existing Models
|
||||
- **Recurrent networks**: Inherently sequential computation
|
||||
- **Convolutional networks**: Limited long-range connectivity
|
||||
- **Hybrid models**: Still rely on recurrence/convolution as primary components
|
||||
|
||||
### Attention Mechanisms
|
||||
- Already used as auxiliary component in sequence models
|
||||
- Allows modeling dependencies without regard to position distance
|
||||
- Not yet used as the primary architectural component
|
||||
|
||||
---
|
||||
|
||||
## The Transformer: Model Architecture
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-76 mx-auto" />
|
||||
|
||||
*Encoder-decoder structure using stacked self-attention and feed-forward layers*
|
||||
|
||||
---
|
||||
|
||||
## Encoder Architecture
|
||||
|
||||
- **Stack of 6 identical layers**
|
||||
- Each layer contains two sub-layers:
|
||||
1. Multi-head self-attention mechanism
|
||||
2. Position-wise fully connected feed-forward network
|
||||
- Residual connections around each sub-layer
|
||||
- Layer normalization: `LayerNorm(x + Sublayer(x))`
|
||||
- Output dimension: `d_model = 512`
|
||||
|
||||
---
|
||||
|
||||
## Decoder Architecture
|
||||
|
||||
- **Stack of 6 identical layers**
|
||||
- Each layer contains three sub-layers:
|
||||
1. Masked multi-head self-attention
|
||||
2. Multi-head attention over encoder output
|
||||
3. Position-wise fully connected feed-forward network
|
||||
- Residual connections and layer normalization
|
||||
- Masking prevents attending to future positions
|
||||
|
||||
---
|
||||
|
||||
## Attention Mechanism Basics
|
||||
|
||||
### Definition
|
||||
An attention function maps a query and key-value pairs to an output:
|
||||
- Query (Q), Keys (K), Values (V) are vectors
|
||||
- Output is weighted sum of values
|
||||
- Weights determined by compatibility of query with corresponding key
|
||||
|
||||
### Two Common Types
|
||||
- Additive attention (feed-forward network)
|
||||
- Dot-product attention (scaled in Transformer)
|
||||
|
||||
---
|
||||
|
||||
## Scaled Dot-Product Attention
|
||||
|
||||
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
|
||||
|
||||
- Computes dot products of query with all keys
|
||||
- Scales by $\sqrt{d_k}$ to prevent gradient vanishing
|
||||
- Applies softmax to get weights over values
|
||||
- More efficient than additive attention (matrix operations)
|
||||
|
||||
---
|
||||
|
||||
## Multi-Head Attention
|
||||
@@ -0,0 +1,30 @@
|
||||
{
|
||||
"overall_score": 85,
|
||||
"pass": false,
|
||||
"issues": [
|
||||
{
|
||||
"page": 4,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "medium",
|
||||
"suggestion": "将6个要点拆分为两页,Recurrent Models和Convolutional Models各占一页,或删减次要内容使每页不超过5个要点"
|
||||
},
|
||||
{
|
||||
"page": 6,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "high",
|
||||
"suggestion": "移除图片下方的大段文字说明,仅保留模型图;文字内容应拆分到后续页面详细讲解"
|
||||
},
|
||||
{
|
||||
"page": 7,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "low",
|
||||
"suggestion": "将\"Each layer contains two sub-layers\"的两个子要点合并为一条,或调整字号增大留白"
|
||||
},
|
||||
{
|
||||
"page": 9,
|
||||
"issue_type": "overcrowded",
|
||||
"severity": "low",
|
||||
"suggestion": "将三个子要点简化为更简短的表述,或拆分到新页面单独说明Decoder的子层结构"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,126 @@
|
||||
---
|
||||
theme: default
|
||||
title: "Attention Is All You Need"
|
||||
author: "Ashish Vaswani et al."
|
||||
---
|
||||
|
||||
# Attention Is All You Need
|
||||
|
||||
**Authors:** Ashish Vaswani et al.
|
||||
**Conference:** NIPS 2017
|
||||
|
||||
---
|
||||
|
||||
## Abstract: Key Innovation
|
||||
|
||||
- Proposes **Transformer** - first architecture based solely on attention
|
||||
- Dispenses with recurrence and convolutions entirely
|
||||
- More parallelizable and requires significantly less training time
|
||||
- Generalizes well to other tasks beyond machine translation
|
||||
|
||||
---
|
||||
|
||||
## Abstract: Performance Highlights
|
||||
|
||||
- **WMT 2014 English-to-German**: 28.4 BLEU
|
||||
- Improves over existing best results by over 2 BLEU
|
||||
- **WMT 2014 English-to-French**: 41.8 BLEU
|
||||
- New single-model state-of-the-art
|
||||
- Trained for 3.5 days on eight GPUs (small fraction of previous costs)
|
||||
|
||||
---
|
||||
|
||||
## Background: Traditional Sequence Modeling
|
||||
|
||||
### Recurrent Models
|
||||
- LSTM and GRU as state-of-the-art approaches
|
||||
- Factor computation along symbol positions
|
||||
- Inherently sequential nature precludes parallelization
|
||||
|
||||
### Convolutional Models
|
||||
- Fixed receptive fields limit long-range dependencies
|
||||
- Computational complexity grows with distance between positions
|
||||
|
||||
---
|
||||
|
||||
## Limitations of Existing Approaches
|
||||
|
||||
- **Recurrent networks**: Sequential computation limits parallelization
|
||||
- **Convolutional networks**: Difficulty modeling long-range dependencies
|
||||
- **Hybrid models**: Still rely on recurrence/convolution as primary components
|
||||
- **Attention mechanisms**: Previously used only as auxiliary component
|
||||
|
||||
---
|
||||
|
||||
## The Transformer: Model Architecture
|
||||
|
||||
<img src="/paper_figure_1_transformer.png" class="h-80 mx-auto" />
|
||||
|
||||
---
|
||||
|
||||
## Encoder Architecture
|
||||
|
||||
- **Stack of 6 identical layers**
|
||||
- Each layer contains two sub-layers:
|
||||
1. Multi-head self-attention mechanism
|
||||
2. Position-wise fully connected feed-forward network
|
||||
- Residual connections around each sub-layer
|
||||
- Layer normalization: `LayerNorm(x + Sublayer(x))`
|
||||
|
||||
---
|
||||
|
||||
## Encoder: Key Details
|
||||
|
||||
- All sub-layers produce outputs of dimension `d_model = 512`
|
||||
- Self-attention allows each position to attend to all positions
|
||||
- Feed-forward network applied to each position separately
|
||||
- Residual connections help with gradient flow in deep networks
|
||||
|
||||
---
|
||||
|
||||
## Decoder Architecture
|
||||
|
||||
- **Stack of 6 identical layers**
|
||||
- Each layer contains three sub-layers:
|
||||
1. Masked multi-head self-attention
|
||||
2. Multi-head attention over encoder output
|
||||
3. Position-wise fully connected feed-forward network
|
||||
- Residual connections and layer normalization
|
||||
|
||||
---
|
||||
|
||||
## Decoder: Masking Mechanism
|
||||
|
||||
- **Masked self-attention** prevents positions from attending to subsequent positions
|
||||
- Ensures predictions for position i depend only on:
|
||||
- Known outputs at positions less than i
|
||||
- Input sequence through encoder-decoder attention
|
||||
- Output embeddings offset by one position to maintain auto-regressive property
|
||||
|
||||
---
|
||||
|
||||
## Attention Mechanism: Definition
|
||||
|
||||
Maps a query and key-value pairs to an output:
|
||||
- Query (Q), Keys (K), Values (V) are vectors
|
||||
- Output is weighted sum of values
|
||||
- Weights determined by query-key compatibility
|
||||
|
||||
---
|
||||
|
||||
## Scaled Dot-Product Attention
|
||||
|
||||
$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$
|
||||
|
||||
- Computes dot products of query with all keys
|
||||
- Scales by $\sqrt{d_k}$ to prevent gradient vanishing
|
||||
- Applies softmax to get weights over values
|
||||
- More efficient than additive attention
|
||||
|
||||
---
|
||||
|
||||
## Multi-Head Attention
|
||||
|
||||
- Projects Q, K, V h times with different learned projections
|
||||
- Performs attention in parallel on each projected version
|
||||
- Concatenates outputs and applies final projection
|
||||
|
After Width: | Height: | Size: 43 KiB |
|
After Width: | Height: | Size: 18 KiB |
|
After Width: | Height: | Size: 94 KiB |
|
After Width: | Height: | Size: 85 KiB |
|
After Width: | Height: | Size: 117 KiB |
|
After Width: | Height: | Size: 122 KiB |
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 82 KiB |
|
After Width: | Height: | Size: 99 KiB |
|
After Width: | Height: | Size: 85 KiB |
|
After Width: | Height: | Size: 43 KiB |
|
After Width: | Height: | Size: 81 KiB |
|
After Width: | Height: | Size: 63 KiB |
|
After Width: | Height: | Size: 82 KiB |
|
After Width: | Height: | Size: 50 KiB |
|
After Width: | Height: | Size: 68 KiB |
|
After Width: | Height: | Size: 76 KiB |
|
After Width: | Height: | Size: 92 KiB |
|
After Width: | Height: | Size: 80 KiB |
|
After Width: | Height: | Size: 100 KiB |
|
After Width: | Height: | Size: 74 KiB |
|
After Width: | Height: | Size: 67 KiB |
|
After Width: | Height: | Size: 72 KiB |
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"paper": {
|
||||
"title": "Attention Is All You Need",
|
||||
"authors": "Ashish Vaswani et al.",
|
||||
"arxiv_id": "1706.03762",
|
||||
"pdf_url": "https://arxiv.org/pdf/1706.03762",
|
||||
"pdf_sha256": "bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697",
|
||||
"observed_pdf_sha256": "bdfaa68d8984f0dc02beaca527b76f207d99b666d31d1da728ee0728182df697"
|
||||
},
|
||||
"paper_text": {
|
||||
"path": "/Users/boj/book/ai-agent-book/chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v3/source/paper_text.md",
|
||||
"sha256": "4da5f2a1da38bad8149832b5ada3c94f0619188db8fed5cdca8ff63405587eb8",
|
||||
"characters": 39819
|
||||
},
|
||||
"visuals": [
|
||||
{
|
||||
"filename": "paper_figure_1_transformer.png",
|
||||
"pdf_page": 3,
|
||||
"source_label": "Figure 1",
|
||||
"caption": "The Transformer model architecture.",
|
||||
"rect": [
|
||||
92,
|
||||
28,
|
||||
520,
|
||||
535
|
||||
],
|
||||
"sha256": "48f09e2e2426e8ca76c1e438600438dca2f9e61360e92ae3edbb538a2cc611d2",
|
||||
"bytes": 150087,
|
||||
"public_copy_sha256": "48f09e2e2426e8ca76c1e438600438dca2f9e61360e92ae3edbb538a2cc611d2"
|
||||
},
|
||||
{
|
||||
"filename": "paper_figure_3_long_distance.png",
|
||||
"pdf_page": 13,
|
||||
"source_label": "Figure 3",
|
||||
"caption": "Encoder self-attention following long-distance dependencies.",
|
||||
"rect": [
|
||||
92,
|
||||
55,
|
||||
525,
|
||||
455
|
||||
],
|
||||
"sha256": "ea456861962564d2ef5a158e4205cfea29b537f4a611df989db5d65471f23d78",
|
||||
"bytes": 96204,
|
||||
"public_copy_sha256": "ea456861962564d2ef5a158e4205cfea29b537f4a611df989db5d65471f23d78"
|
||||
},
|
||||
{
|
||||
"filename": "paper_figure_4_anaphora.png",
|
||||
"pdf_page": 14,
|
||||
"source_label": "Figure 4",
|
||||
"caption": "Attention heads involved in anaphora resolution.",
|
||||
"rect": [
|
||||
92,
|
||||
135,
|
||||
525,
|
||||
665
|
||||
],
|
||||
"sha256": "b1f0c6b0e186acb50cae69c304a9a2a0e1f09fd15f1beb0bba03194528163d7e",
|
||||
"bytes": 264855,
|
||||
"public_copy_sha256": "b1f0c6b0e186acb50cae69c304a9a2a0e1f09fd15f1beb0bba03194528163d7e"
|
||||
}
|
||||
]
|
||||
}
|
||||
|
After Width: | Height: | Size: 147 KiB |
|
After Width: | Height: | Size: 94 KiB |
|
After Width: | Height: | Size: 259 KiB |