ai-agent-book/chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-both-20260730-v4/dual_round1_slides.md

theme, title, author, date

theme	title	author	date
default	Attention Is All You Need	Ashish Vaswani et al.	NIPS 2017

Attention Is All You Need

Authors: Ashish Vaswani et al.
Conference: NIPS 2017

---

Abstract

• Proposes Transformer - a new network architecture based solely on attention
• Dispenses with recurrence and convolutions entirely
• More parallelizable and requires significantly less training time
• State-of-the-art results:
  • 28.4 BLEU on WMT 2014 English-to-German
  • 41.8 BLEU on WMT 2014 English-to-French
• Generalizes well to other tasks like English constituency parsing

---

Background & Key Idea

Traditional Sequence Models

• Recurrent models (LSTM, GRU): Sequential computation, limited parallelization
• Convolutional models: Limited long-range dependencies

Attention Mechanism

• Already used with recurrent networks
• Allows modeling dependencies without regard to distance

The Transformer

• Key innovation: Replace recurrence and convolution with self-attention
• Enables more parallelization and better long-range dependencies

---

Transformer Architecture

Encoder (left) and Decoder (right) stacks with self-attention and feed-forward layers

---

Encoder & Decoder Stacks

Encoder

• 6 identical layers each with:
  1. Multi-head self-attention mechanism
  2. Position-wise fully connected feed-forward network
• Residual connections + layer normalization
• Output dimension: d_model = 512

Decoder

• 6 identical layers each with:
  1. Masked multi-head self-attention (prevents leftward flow)
  2. Multi-head attention over encoder output
  3. Position-wise feed-forward network
• Same residual connections and layer normalization

---

Attention Mechanism

Scaled Dot-Product Attention

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

• Q (queries), K (keys), V (values) are matrices
• Scaling by \sqrt{d_k} prevents gradients from becoming too small
• More efficient than additive attention

---

Multi-Head Attention

• Projects queries, keys, values h times with different linear projections
• Performs attention in parallel on each projected version
• Concatenates results and projects again:

\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O

• h = 8 heads, d_k = d_v = d_{\text{model}}/h = 64

Applications

1. Encoder-decoder attention: Queries from decoder, keys/values from encoder
2. Encoder self-attention: All positions attend to all positions
3. Decoder self-attention: Positions attend to previous positions (masked)

---

Positional Encoding

Since there's no recurrence/convolution, we add positional information:

\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right) \text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)

• Same dimension as embeddings (d_{\text{model}})
• Allows model to learn relative position information
• Performed similarly to learned positional embeddings

---

Why Self-Attention?

Layer Type	Complexity	Sequential Operations	Max Path Length
Self-Attention	O(n^2 \cdot d)	O(1)	O(1)
Recurrent	O(n \cdot d^2)	O(n)	O(n)
Convolutional	O(k \cdot n \cdot d^2)	O(1)	O(\log_k n)

• Better parallelization than RNNs
• Shorter path length than CNNs/RNNs
• More interpretable attention distributions

---

Training Details

Data & Batching

• WMT 2014 English-German (4.5M pairs), English-French (36M pairs)
• Byte-pair encoding with shared vocabulary
• Batches with ~25,000 source and target tokens

Hardware & Schedule

• 8 NVIDIA P100 GPUs
• Base model: 100,000 steps (12 hours)
• Big model: 300,000 steps (3.5 days)

Optimizer & Regularization

• Adam with \beta_1=0.9, \beta_2=0.98, \epsilon=10^{-9}
• Learning rate schedule with warmup steps=4000
• Residual dropout (P_drop=0.1) and label smoothing (\epsilon_{ls}=0.1)

---

Machine Translation Results

Model	EN-DE BLEU	EN-FR BLEU	Training Cost (FLOPs)
GNMT + RL Ensemble	26.30	41.16	1.8 \cdot 10^{20}
ConvS2S Ensemble	26.36	41.29	7.7 \cdot 10^{19}
Transformer (big)	28.4	41.8	$2.3 \cdot 10^{19}$

• Transformer outperforms all previous state-of-the-art models
• Achieves better results with significantly lower training cost
• 2+ BLEU improvement on English-to-German

---

Model Architecture Ablations

Variation	Dev PPL	Dev BLEU
Base model	4.92	25.8
(A) 1 attention head	5.29	24.9
(B) d_k = 16	5.75	24.5
(C) 2 layers	6.11	23.7
(D) No dropout	5.77	24.6
(E) Learned positional embeddings	4.92	25.7

• Multi-head attention improves performance
• Dropout is crucial for avoiding overfitting
• Sinusoidal and learned positional encodings perform similarly

---

English Constituency Parsing

Parser	Training	WSJ 23 F1
Petrov et al. (2006)	WSJ only	90.4
Dyer et al. (2016)	WSJ only	91.7
Transformer (4 layers)	WSJ only	91.3
Vinyals & Kaiser et al. (2014)	Semi-supervised	92.1
Transformer (4 layers)	Semi-supervised	92.7

• Transformer generalizes well to other sequence tasks
• Performs comparably to state-of-the-art parsers
• Better than RNN models in small-data regimes

---

Attention Visualization: Long-Distance Dependencies

Encoder self-attention in layer 5 showing attention to distant dependency "making...more difficult"

---

Attention Visualization: Anaphora Resolution

Attention heads focusing on resolving "its" to "The Law"

---

Conclusion & Future Work

Contributions

• Introduced Transformer, first transduction model based entirely on attention
• Eliminated recurrence and convolution
• Achieved new state-of-the-art in machine translation
• More parallelizable and faster to train
• Generalizes well to other tasks

Future Work

• Apply to other modalities (images, audio, video)
• Investigate local, restricted attention mechanisms
• Make generation less sequential
• Explore interpretability of attention patterns

Code available at: https://github.com/tensorflow/tensor2tensor