ai-agent-book 精选快照（<2MB 代码与文档，来自 github.com/bojieli/ai-agent-book）
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
Build latest book artifacts / build (push) Canceled after 0s
dependency resolution / resolve (3.11) (push) Canceled after 0s
dependency resolution / resolve (3.13) (push) Canceled after 0s
deploy-pages / build (push) Canceled after 0s
deploy-pages / deploy (push) Canceled after 0s
i18n consistency check / check (push) Canceled after 0s
provider adoption tests / test (chapter2/context-compression) (push) Canceled after 0s
provider adoption tests / test (chapter2/prompt-injection) (push) Canceled after 0s
provider adoption tests / test (chapter2/system-hint) (push) Canceled after 0s
provider adoption tests / test (chapter3/log-sanitization) (push) Canceled after 0s
web-search-agent tests / test (push) Canceled after 0s
web-search-agent tests / agentbook (push) Canceled after 0s
This commit is contained in:
+31
@@ -0,0 +1,31 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #14102E; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; font-family: Arial, sans-serif; display: flex; background-image: url('../assets/cover-bg.png'); }
|
||||
.left { width: 440pt; margin: 44pt 0 44pt 48pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 10pt; color: #F59E0B; font-weight: bold; letter-spacing: 2pt; margin: 0 0 14pt 0; }
|
||||
h1 { font-size: 40pt; color: #FFFFFF; margin: 0 0 14pt 0; line-height: 1.1; }
|
||||
.sub { font-size: 14pt; color: #DDD6FE; margin: 0 0 26pt 0; line-height: 1.35; }
|
||||
.rule { background: #F59E0B; height: 4pt; width: 72pt; margin: 0 0 26pt 0; }
|
||||
.authors { font-size: 11pt; color: #EDE9FE; margin: 0 0 6pt 0; line-height: 1.4; }
|
||||
.affil { font-size: 9.5pt; color: #B8AEE8; margin: 0; }
|
||||
.card { width: 156pt; margin: 46pt 0 0 16pt; background: #FFFFFF; border-radius: 10pt; padding: 12pt; box-shadow: 4px 4px 18px rgba(0,0,0,0.4); height: 256pt; }
|
||||
.card img { height: 200pt; display: block; margin: 0 auto; }
|
||||
.cardcap { font-size: 8pt; color: #5B5876; text-align: center; margin: 8pt 0 0 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="left">
|
||||
<p class="kicker">NEURIPS 2017 · ARXIV:1706.03762</p>
|
||||
<h1>Attention Is<br>All You Need</h1>
|
||||
<p class="sub">The Transformer: sequence transduction based entirely on attention — no recurrence, no convolution.</p>
|
||||
<div class="rule"></div>
|
||||
<p class="authors">Ashish Vaswani · Noam Shazeer · Niki Parmar · Jakob Uszkoreit<br>Llion Jones · Aidan N. Gomez · Łukasz Kaiser · Illia Polosukhin</p>
|
||||
<p class="affil">Google Brain · Google Research · University of Toronto</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<img src="../source_visuals/fig1-03.png">
|
||||
<p class="cardcap">The Transformer — model architecture<br>Figure 1, paper p. 3</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+46
@@ -0,0 +1,46 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 14pt 0; }
|
||||
.cols { display: flex; }
|
||||
.txt { width: 400pt; }
|
||||
ul { margin: 0; padding-left: 14pt; font-size: 11pt; color: #2A2740; line-height: 1.4; }
|
||||
li { margin-bottom: 9pt; }
|
||||
.stat { width: 200pt; margin-left: 30pt; background: #EEF0FF; border-radius: 10pt; padding: 14pt 16pt; }
|
||||
.stat p { margin: 0; }
|
||||
.bignum { font-size: 26pt; color: #4338CA; font-weight: bold; margin: 0; }
|
||||
.statlab { font-size: 9.5pt; color: #5B5876; margin: 2pt 0 12pt 0; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">01 · BACKGROUND</p>
|
||||
<h1>The Problem: The Limits of Recurrence</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="txt">
|
||||
<ul>
|
||||
<li>Sequence transduction (e.g., machine translation) was dominated by <b>recurrent</b> (LSTM / GRU) and <b>convolutional</b> encoder–decoder models.</li>
|
||||
<li>RNNs compute hidden states <b>one position at a time</b>: inherently sequential, so training cannot parallelize across positions in a sequence.</li>
|
||||
<li>Signals between distant positions must travel a path of <b>O(n) operations</b> — the longer the path, the harder long-range dependencies are to learn.</li>
|
||||
<li>Attention mechanisms already existed, but only as an <b>accessory</b> layered on top of RNNs or CNNs.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<p class="bignum">O(n)</p>
|
||||
<p class="statlab">sequential operations per layer in a recurrent network</p>
|
||||
<p class="bignum">O(n)</p>
|
||||
<p class="statlab">maximum path length between any two positions</p>
|
||||
</div>
|
||||
</div>
|
||||
<p class="foot">Source: Vaswani et al., “Attention Is All You Need”, §1 Introduction (paper p. 1–2).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 16pt 0; }
|
||||
.cards { display: flex; }
|
||||
.card { width: 198pt; margin-right: 14pt; background: #4338CA; border-radius: 10pt; padding: 14pt 14pt 16pt 14pt; }
|
||||
.card p { margin: 0; }
|
||||
.ct { font-size: 13pt; color: #FFFFFF; font-weight: bold; margin: 0 0 6pt 0; }
|
||||
.cd { font-size: 9.5pt; color: #DDD6FE; line-height: 1.35; }
|
||||
ul { margin: 16pt 0 0 0; padding-left: 14pt; font-size: 11pt; color: #2A2740; line-height: 1.4; }
|
||||
li { margin-bottom: 8pt; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">02 · KEY IDEA</p>
|
||||
<h1>The Transformer: Attention Is All You Need</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cards">
|
||||
<div class="card">
|
||||
<p class="ct">No Recurrence</p>
|
||||
<p class="cd">Sequential RNN hidden states are removed entirely — positions are processed in parallel.</p>
|
||||
</div>
|
||||
<div class="card">
|
||||
<p class="ct">No Convolution</p>
|
||||
<p class="cd">No fixed-width kernels; every position can connect to every other position directly.</p>
|
||||
</div>
|
||||
<div class="card" style="margin-right: 0;">
|
||||
<p class="ct">Self-Attention Only</p>
|
||||
<p class="cd">Multi-headed self-attention computes all representations, in both encoder and decoder.</p>
|
||||
</div>
|
||||
</div>
|
||||
<ul>
|
||||
<li>First sequence transduction model based <b>entirely on attention</b>, replacing recurrent layers with multi-headed self-attention.</li>
|
||||
<li>Highly parallelizable — reaches a new state of the art in translation with a <b>fraction of the training cost</b>.</li>
|
||||
<li>Generalizes beyond translation (e.g., English constituency parsing) and yields interpretable attention patterns.</li>
|
||||
</ul>
|
||||
<p class="foot">Source: Vaswani et al., Abstract and §1 (paper p. 1–2).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 12pt 0; }
|
||||
.cols { display: flex; }
|
||||
.txt { width: 372pt; }
|
||||
ul { margin: 0; padding-left: 14pt; font-size: 10.5pt; color: #2A2740; line-height: 1.38; }
|
||||
li { margin-bottom: 8pt; }
|
||||
.fig { width: 240pt; margin-left: 26pt; background: #FFFFFF; border: 1px solid #E0DFF0; border-radius: 10pt; padding: 10pt; }
|
||||
.fig img { height: 262pt; display: block; margin: 0 auto; }
|
||||
.figcap { font-size: 8pt; color: #5B5876; text-align: center; margin: 6pt 0 0 0; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">03 · METHOD</p>
|
||||
<h1>Model Architecture: Stacked Self-Attention</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="txt">
|
||||
<ul>
|
||||
<li><b>Encoder–decoder</b> structure built from stacked self-attention and point-wise, fully connected layers (left and right halves of Figure 1).</li>
|
||||
<li><b>Encoder</b>: N = 6 identical layers — each with multi-head self-attention, then a position-wise feed-forward network.</li>
|
||||
<li><b>Decoder</b>: N = 6 layers; adds a <b>masked</b> multi-head self-attention sub-layer (positions only attend to earlier positions) plus attention over the encoder output.</li>
|
||||
<li>A <b>residual connection</b> wraps every sub-layer, followed by <b>layer normalization</b>.</li>
|
||||
<li>Inputs/outputs are embedded (d_model = 512) and combined with <b>positional encodings</b>, since the model has no notion of order on its own.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="fig">
|
||||
<img src="../source_visuals/fig1-03.png">
|
||||
<p class="figcap">Encoder (left) and decoder (right) stacks.<br>Figure 1, paper p. 3.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p class="foot">Visual: Figure 1, “The Transformer - model architecture” (paper p. 3); text: §3.1 (paper p. 3).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+48
@@ -0,0 +1,48 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 12pt 0; }
|
||||
.cols { display: flex; }
|
||||
.txt { width: 300pt; }
|
||||
.formula { background: #1A1633; border-radius: 8pt; padding: 10pt 12pt; margin: 0 0 12pt 0; }
|
||||
.formula p { font-family: Courier New, monospace; font-size: 10.5pt; color: #FDE68A; margin: 0; text-align: center; }
|
||||
ul { margin: 0; padding-left: 14pt; font-size: 10.5pt; color: #2A2740; line-height: 1.38; }
|
||||
li { margin-bottom: 8pt; }
|
||||
.fig { width: 312pt; margin-left: 24pt; background: #FFFFFF; border: 1px solid #E0DFF0; border-radius: 10pt; padding: 10pt; }
|
||||
.fig img { width: 292pt; display: block; margin: 0 auto; }
|
||||
.figcap { font-size: 8pt; color: #5B5876; text-align: center; margin: 6pt 0 0 0; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">04 · METHOD</p>
|
||||
<h1>Scaled Dot-Product & Multi-Head Attention</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="txt">
|
||||
<div class="formula">
|
||||
<p>Attention(Q,K,V) = softmax(QK^T / √d_k)V</p>
|
||||
</div>
|
||||
<ul>
|
||||
<li>Attention maps a <b>query</b> and <b>key–value</b> pairs to an output: a weighted sum of values, weights from query–key compatibility.</li>
|
||||
<li>Dot products are scaled by <b>1/√d_k</b> — for large d_k they would push softmax into tiny-gradient regions.</li>
|
||||
<li><b>Multi-head</b>: queries, keys, values are linearly projected <b>h = 8</b> times (to d_k = d_v = 64), attention runs in parallel, outputs are concatenated and re-projected.</li>
|
||||
<li>Different heads jointly attend to information from <b>different representation subspaces</b> at different positions.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="fig">
|
||||
<img src="../source_visuals/fig2-04.png">
|
||||
<p class="figcap">(left) Scaled Dot-Product Attention; (right) Multi-Head Attention — several attention layers in parallel. Figure 2, paper p. 4.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p class="foot">Visual: Figure 2 (paper p. 4); formula and text: §3.2 (paper p. 4–5).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+55
@@ -0,0 +1,55 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 14pt 0; }
|
||||
.grid { display: flex; margin-bottom: 12pt; }
|
||||
.card { width: 300pt; background: #FFFFFF; border: 1px solid #E0DFF0; border-left: 5pt solid #7C3AED; border-radius: 8pt; padding: 10pt 14pt; }
|
||||
.card p { margin: 0; }
|
||||
.ct { font-size: 12pt; color: #4338CA; font-weight: bold; margin: 0 0 5pt 0; }
|
||||
.cd { font-size: 10pt; color: #2A2740; line-height: 1.35; }
|
||||
.gap { width: 16pt; }
|
||||
.note { background: #FDF3E0; border-left: 5pt solid #F59E0B; border-radius: 6pt; padding: 9pt 14pt; margin-top: auto; }
|
||||
.note p { font-size: 10pt; color: #6B4E12; margin: 0; line-height: 1.35; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin: 8pt 0 0 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">05 · METHOD</p>
|
||||
<h1>Anatomy of the Building Blocks</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="grid">
|
||||
<div class="card">
|
||||
<p class="ct">Multi-Head Attention</p>
|
||||
<p class="cd">h = 8 parallel heads; d_k = d_v = d_model/h = 64, so total compute stays close to single-head attention at full dimension.</p>
|
||||
</div>
|
||||
<div class="gap"></div>
|
||||
<div class="card">
|
||||
<p class="ct">Position-wise Feed-Forward</p>
|
||||
<p class="cd">FFN(x) = max(0, xW₁+b₁)W₂+b₂, applied identically to every position; inner dimension d_ff = 2048.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="grid">
|
||||
<div class="card">
|
||||
<p class="ct">Add & Norm</p>
|
||||
<p class="cd">Residual connection around each sub-layer: LayerNorm(x + Sublayer(x)); all sub-layers and embeddings output d_model = 512.</p>
|
||||
</div>
|
||||
<div class="gap"></div>
|
||||
<div class="card">
|
||||
<p class="ct">Positional Encoding</p>
|
||||
<p class="cd">Sine/cosine waves of different frequencies injected at the inputs, so the model can use token order; learned alternatives worked equally well.</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="note">
|
||||
<p><b>Two configurations:</b> base — d_model 512, d_ff 2048, h 8, dropout 0.1 | big — d_model 1024, d_ff 4096, h 16, dropout 0.3.</p>
|
||||
</div>
|
||||
<p class="foot">Source: §3.2–3.5 (paper p. 4–5); §5.4 and Table 3 footnotes (paper p. 7–8).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+44
@@ -0,0 +1,44 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 12pt 0; }
|
||||
.cols { display: flex; }
|
||||
.txt { width: 290pt; }
|
||||
ul { margin: 0; padding-left: 14pt; font-size: 10.5pt; color: #2A2740; line-height: 1.38; }
|
||||
li { margin-bottom: 9pt; }
|
||||
.tabwrap { width: 330pt; margin-left: 22pt; }
|
||||
.tablab { font-size: 9pt; color: #5B5876; margin: 0 0 4pt 0; font-weight: bold; }
|
||||
.tabcap { font-size: 8pt; color: #5B5876; margin: 6pt 0 0 0; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">06 · MOTIVATION</p>
|
||||
<h1>Why Self-Attention Wins</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="txt">
|
||||
<ul>
|
||||
<li><b>Parallelization:</b> a self-attention layer needs only O(1) sequential operations vs. O(n) for recurrence — training uses all positions at once.</li>
|
||||
<li><b>Long-range learning:</b> any two positions connect through a path of <b>constant length O(1)</b>, instead of O(n) for RNNs or O(log_k n) for dilated convolutions.</li>
|
||||
<li><b>Cost:</b> per-layer complexity O(n²·d) beats recurrence O(n·d²) whenever n < d — the common case with word-piece / BPE representations.</li>
|
||||
<li><b>Interpretability:</b> attention distributions expose what the model attends to; heads learn distinct, task-like behaviors.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="tabwrap">
|
||||
<p class="tablab">PER-LAYER COMPLEXITY AND PATH LENGTHS</p>
|
||||
<div id="table-complexity" class="placeholder" style="width: 330pt; height: 150pt;"></div>
|
||||
<p class="tabcap">n = sequence length, d = representation dimension, k = conv. kernel width, r = neighborhood. Adapted from Table 1, paper p. 6.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p class="foot">Table: adapted from Table 1 (paper p. 6); text: §4 (paper p. 6–7).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+54
@@ -0,0 +1,54 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 14pt 0; }
|
||||
.cols { display: flex; }
|
||||
.card { width: 302pt; background: #FFFFFF; border: 1px solid #E0DFF0; border-radius: 10pt; padding: 12pt 16pt 4pt 16pt; }
|
||||
.card p { margin: 0; }
|
||||
.ct { font-size: 12.5pt; color: #4338CA; font-weight: bold; margin: 0 0 8pt 0; }
|
||||
ul { margin: 0; padding-left: 13pt; font-size: 10pt; color: #2A2740; line-height: 1.35; }
|
||||
li { margin-bottom: 7pt; }
|
||||
.gap { width: 20pt; }
|
||||
.mono { font-family: Courier New, monospace; font-size: 9pt; color: #4338CA; }
|
||||
.hw { background: #EEF0FF; border-radius: 8pt; padding: 9pt 14pt; margin-top: 14pt; }
|
||||
.hw p { font-size: 10pt; color: #2A2740; margin: 0; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">07 · EXPERIMENTS</p>
|
||||
<h1>Training Setup</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="card">
|
||||
<p class="ct">Data & Batching</p>
|
||||
<ul>
|
||||
<li><b>WMT 2014 English–German</b>: 4.5M sentence pairs; <b>English–French</b>: 36M sentences.</li>
|
||||
<li>Byte-pair encoding with a <b>shared source–target vocabulary of ~37K tokens</b>.</li>
|
||||
<li>Batches of ~25,000 source and ~25,000 target tokens, grouped by approximate sequence length.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="gap"></div>
|
||||
<div class="card">
|
||||
<p class="ct">Optimization & Regularization</p>
|
||||
<ul>
|
||||
<li>Adam (β₁ = 0.9, β₂ = 0.98, ε = 10⁻⁹) with <b>warmup</b>: 4,000 linearly increasing steps, then inverse-square-root decay.</li>
|
||||
<li>Residual dropout 0.1 (base) / 0.3 (big); attention dropout on each sub-layer.</li>
|
||||
<li>Label smoothing ε_ls = 0.1 — hurts perplexity but <b>improves accuracy and BLEU</b>.</li>
|
||||
</ul>
|
||||
</div>
|
||||
</div>
|
||||
<div class="hw">
|
||||
<p><b>Hardware & time:</b> 8 × NVIDIA P100 GPUs — base model: 12 hours (100K steps); big model: 3.5 days (300K steps). Beam search with beam size 4, length penalty α = 0.6.</p>
|
||||
</div>
|
||||
<p class="foot">Source: §5 Training (paper p. 7); §6.1 (paper p. 8).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+52
@@ -0,0 +1,52 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 12pt 0; }
|
||||
.cols { display: flex; }
|
||||
.fig { width: 396pt; background: #FFFFFF; border: 1px solid #E0DFF0; border-radius: 10pt; padding: 10pt; }
|
||||
.fig img { width: 376pt; display: block; margin: 0 auto; }
|
||||
.figcap { font-size: 8pt; color: #5B5876; text-align: center; margin: 6pt 0 0 0; }
|
||||
.stats { width: 196pt; margin-left: 22pt; }
|
||||
.stat { background: #4338CA; border-radius: 10pt; padding: 10pt 14pt; margin-bottom: 10pt; }
|
||||
.stat p { margin: 0; }
|
||||
.bignum { font-size: 22pt; color: #FFFFFF; font-weight: bold; }
|
||||
.statlab { font-size: 8.5pt; color: #DDD6FE; line-height: 1.3; margin-top: 2pt; }
|
||||
.stat.amber { background: #B45309; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">08 · RESULTS</p>
|
||||
<h1>Machine Translation: New State of the Art</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="fig">
|
||||
<img src="../source_visuals/table2-08.png">
|
||||
<p class="figcap">BLEU scores and training costs on WMT 2014 EN–DE / EN–FR (newstest2014). Table 2, paper p. 8.</p>
|
||||
</div>
|
||||
<div class="stats">
|
||||
<div class="stat">
|
||||
<p class="bignum">28.4 BLEU</p>
|
||||
<p class="statlab">EN→DE: >2.0 above the best previously reported model, including ensembles</p>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<p class="bignum">41.8 BLEU</p>
|
||||
<p class="statlab">EN→FR: best single model, at under 1/4 of the prior training cost</p>
|
||||
</div>
|
||||
<div class="stat amber">
|
||||
<p class="bignum">3.3×10^18</p>
|
||||
<p class="statlab">FLOPs to train the base model — yet it already beats all prior models and ensembles</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p class="foot">Visual: Table 2 (paper p. 8); text: §6.1 Machine Translation (paper p. 8).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+34
@@ -0,0 +1,34 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 10pt 0; }
|
||||
.chartwrap { margin: 0 auto; }
|
||||
#chart-bleu { margin: 0 auto; }
|
||||
.chartcap { font-size: 9pt; color: #5B5876; text-align: center; margin: 6pt 0 0 0; }
|
||||
.takeaway { background: #FDF3E0; border-left: 5pt solid #F59E0B; border-radius: 6pt; padding: 8pt 14pt; margin-top: auto; }
|
||||
.takeaway p { font-size: 10pt; color: #6B4E12; margin: 0; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin: 8pt 0 0 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">09 · RESULTS</p>
|
||||
<h1>EN→DE BLEU: Transformer vs. Prior Art</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="chartwrap">
|
||||
<div id="chart-bleu" class="placeholder" style="width: 460pt; height: 218pt; margin: 0 auto;"></div>
|
||||
<p class="chartcap">WMT 2014 English→German newstest2014 BLEU scores — values from Table 2, paper p. 8.</p>
|
||||
</div>
|
||||
<div class="takeaway">
|
||||
<p>The big Transformer sets a new state of the art (28.4 BLEU) — more than 2.0 BLEU above the best previously reported models, including ensembles.</p>
|
||||
</div>
|
||||
<p class="foot">Data: Table 2 (paper p. 8); discussion: §6.1 (paper p. 8).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+51
@@ -0,0 +1,51 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 14pt 0; }
|
||||
.cols { display: flex; }
|
||||
.txt { width: 380pt; }
|
||||
ul { margin: 0; padding-left: 14pt; font-size: 11pt; color: #2A2740; line-height: 1.4; }
|
||||
li { margin-bottom: 10pt; }
|
||||
.stats { width: 210pt; margin-left: 30pt; }
|
||||
.stat { background: #4338CA; border-radius: 10pt; padding: 12pt 16pt; margin-bottom: 12pt; }
|
||||
.stat p { margin: 0; }
|
||||
.bignum { font-size: 24pt; color: #FFFFFF; font-weight: bold; }
|
||||
.statlab { font-size: 9pt; color: #DDD6FE; line-height: 1.3; margin-top: 3pt; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">10 · RESULTS</p>
|
||||
<h1>Beyond Translation: Constituency Parsing</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="txt">
|
||||
<ul>
|
||||
<li>A <b>4-layer Transformer</b> was trained on English constituency parsing (Wall Street Journal portion of the Penn Treebank, ~40K training sentences).</li>
|
||||
<li>With almost <b>no task-specific tuning</b>, it outperforms the BerkeleyParser — even when trained only on the WSJ training set.</li>
|
||||
<li>In the <b>semi-supervised</b> setting (with a ~17M-sentence high-confidence corpus) it beats all previously reported models <b>except the Recurrent Neural Network Grammar</b>.</li>
|
||||
<li>The same architecture generalizes across tasks without structural changes.</li>
|
||||
</ul>
|
||||
</div>
|
||||
<div class="stats">
|
||||
<div class="stat">
|
||||
<p class="bignum">91.3 F1</p>
|
||||
<p class="statlab">WSJ-only, discriminative setting (Section 23 of WSJ)</p>
|
||||
</div>
|
||||
<div class="stat">
|
||||
<p class="bignum">92.7 F1</p>
|
||||
<p class="statlab">Semi-supervised setting — above every prior model except RNNG</p>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
<p class="foot">Source: Table 4 (paper p. 10) and §6.2 / “English Constituency Parsing” (paper p. 9–10).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+47
@@ -0,0 +1,47 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #FFFFFF; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; background: #FAFAFF; font-family: Arial, sans-serif; display: flex; }
|
||||
.bar { width: 10pt; background: #4338CA; }
|
||||
.wrap { width: 644pt; margin: 26pt 32pt 22pt 34pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #7C3AED; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 23pt; color: #1A1633; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 12pt 0; }
|
||||
.cols { display: flex; }
|
||||
.txt { width: 282pt; }
|
||||
ul { margin: 0; padding-left: 14pt; font-size: 10.5pt; color: #2A2740; line-height: 1.38; }
|
||||
li { margin-bottom: 9pt; }
|
||||
.quote { background: #EEF0FF; border-left: 4pt solid #7C3AED; border-radius: 6pt; padding: 8pt 12pt; margin-top: 10pt; }
|
||||
.quote p { font-size: 9.5pt; color: #4338CA; font-style: italic; margin: 0; line-height: 1.35; }
|
||||
.fig { width: 330pt; margin-left: 24pt; background: #FFFFFF; border: 1px solid #E0DFF0; border-radius: 10pt; padding: 10pt; }
|
||||
.fig img { width: 310pt; display: block; margin: 0 auto; }
|
||||
.figcap { font-size: 8pt; color: #5B5876; text-align: center; margin: 6pt 0 0 0; }
|
||||
.foot { font-size: 8pt; color: #8A86A3; margin-top: auto; margin-bottom: 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="bar"></div>
|
||||
<div class="wrap">
|
||||
<p class="kicker">11 · RESULTS</p>
|
||||
<h1>Interpretability: What Attention Sees</h1>
|
||||
<div class="rule"></div>
|
||||
<div class="cols">
|
||||
<div class="txt">
|
||||
<ul>
|
||||
<li>Individual attention heads learn to perform <b>different tasks</b>; many exhibit behavior tied to the syntactic and semantic structure of sentences.</li>
|
||||
<li>In this example (encoder self-attention, layer 5 of 6), many heads attend to a <b>distant dependency of the verb “making”</b>, completing the phrase “making…more difficult”.</li>
|
||||
<li>Other heads are involved in <b>anaphora resolution</b> (Figure 4 of the paper).</li>
|
||||
</ul>
|
||||
<div class="quote">
|
||||
<p>“Not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences.” — §4, p. 7</p>
|
||||
</div>
|
||||
</div>
|
||||
<div class="fig">
|
||||
<img src="../source_visuals/fig3-13.png">
|
||||
<p class="figcap">Attention for the word “making”; colors = different heads. Figure 3, paper p. 13.</p>
|
||||
</div>
|
||||
</div>
|
||||
<p class="foot">Visual: Figure 3 (paper p. 13); text: §4 (paper p. 7) and Appendix (paper p. 13).</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
<!DOCTYPE html>
|
||||
<html>
|
||||
<head><style>
|
||||
html { background: #14102E; }
|
||||
body { width: 720pt; height: 405pt; margin: 0; padding: 0; font-family: Arial, sans-serif; display: flex; background-image: url('../assets/cover-bg.png'); }
|
||||
.wrap { width: 620pt; margin: 30pt 50pt 24pt 50pt; display: flex; flex-direction: column; }
|
||||
.kicker { font-size: 9.5pt; color: #F59E0B; font-weight: bold; letter-spacing: 2pt; margin: 0 0 4pt 0; }
|
||||
h1 { font-size: 24pt; color: #FFFFFF; margin: 0 0 8pt 0; }
|
||||
.rule { background: #F59E0B; height: 3pt; width: 60pt; margin: 0 0 14pt 0; }
|
||||
ul { margin: 0; padding-left: 15pt; font-size: 11pt; color: #EDE9FE; line-height: 1.42; }
|
||||
li { margin-bottom: 8pt; }
|
||||
.quote { background: #3B2380; border-left: 4pt solid #F59E0B; border-radius: 6pt; padding: 10pt 14pt; margin-top: auto; }
|
||||
.quote p { font-size: 10pt; color: #DDD6FE; font-style: italic; margin: 0; line-height: 1.4; }
|
||||
.foot { font-size: 8pt; color: #B8AEE8; margin: 10pt 0 0 0; }
|
||||
</style></head>
|
||||
<body>
|
||||
<div class="wrap">
|
||||
<p class="kicker">12 · CONCLUSION</p>
|
||||
<h1>Conclusion & Legacy</h1>
|
||||
<div class="rule"></div>
|
||||
<ul>
|
||||
<li>The <b>Transformer</b>: the first sequence transduction model based entirely on attention — multi-headed self-attention replaces recurrent layers in encoder–decoder architectures.</li>
|
||||
<li>Trains <b>significantly faster</b> than recurrent or convolutional architectures thanks to full parallelization.</li>
|
||||
<li>New state of the art on <b>WMT 2014 EN→DE (28.4 BLEU)</b> and <b>EN→FR (41.8 BLEU)</b>, at a fraction of the previous training cost.</li>
|
||||
<li>The architecture became the foundation of modern large language models (BERT, GPT, and their successors) — attention really was all you needed.</li>
|
||||
</ul>
|
||||
<div class="quote">
|
||||
<p>“We are excited about the future of attention-based models and plan to apply them to other tasks.” — Vaswani et al., §7 Conclusion (paper p. 10)</p>
|
||||
</div>
|
||||
<p class="foot">Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, Polosukhin. “Attention Is All You Need.” NeurIPS 2017 (arXiv:1706.03762). All figures/tables shown are cropped from the original paper PDF.</p>
|
||||
</div>
|
||||
</body>
|
||||
</html>
|
||||
Reference in New Issue
Block a user