ai-agent-book/chapter5/paper-to-ppt/validation/runs/exp5-4-real-pdf-20260730-v2/dual_round3_slides.md

theme

theme
default

Attention Is All You Need

A Revolutionary Architecture for Sequence Transduction

Ashish Vaswani et al.
NIPS 2017

---

Abstract: Key Innovation

• Proposes Transformer - first model based solely on attention mechanisms
• Dispenses with recurrence and convolutions entirely
• More parallelizable and requires significantly less training time

---

Abstract: Performance Highlights

• Achieves 28.4 BLEU on WMT 2014 English-to-German
  • Improves over existing best results by over 2 BLEU
• Establishes new state-of-the-art 41.8 BLEU on WMT 2014 English-to-French
• Generalizes well to other tasks like English constituency parsing

---

Background: Limitations of RNNs

Recurrent Neural Networks (RNNs/LSTMs/GRUs)

• Inherently sequential computation
• Cannot parallelize within training examples
• Difficult to learn long-range dependencies
• Memory constraints limit batching for long sequences

---

Background: Limitations of Convolutional Approaches

CNN-based Models (ByteNet, ConvS2S)

• Use convolutions for parallelization
• Number of operations grows with distance between positions
  • Linear growth for ConvS2S
  • Logarithmic growth for ByteNet
• Longer path lengths between distant positions

---

Key Insight: Attention is Sufficient

Attention mechanisms allow modeling dependencies without regard to distance, but were previously used with RNNs.

Transformer: First transduction model relying entirely on self-attention to compute representations without:

• Sequence-aligned RNNs
• Convolutions

---

Transformer Architecture Overview

Encoder-decoder structure with stacked self-attention and feed-forward layers

---

Encoder Structure

• Stack of 6 identical layers
• Residual connections around each sub-layer
• Layer normalization after each sub-layer
• All sub-layers produce outputs of dimension d_model = 512

---

Encoder: Sub-layer Details

Each encoder layer contains two sub-layers:

1. Multi-head self-attention mechanism

   • All positions attend to all positions in previous layer
   • Enables modeling of dependencies throughout sequence

2. Position-wise fully connected feed-forward network

   • Applied to each position separately and identically
   • Two linear transformations with ReLU activation

---

Decoder Structure

• Stack of 6 identical layers
• Residual connections and layer normalization
• Output embeddings offset by one position (auto-regressive property)

---

Decoder: Sub-layer Details

Each decoder layer contains three sub-layers:

1. Masked multi-head self-attention

   • Prevents positions from attending to subsequent positions

2. Multi-head attention over encoder output

   • Queries from decoder, keys/values from encoder

3. Position-wise fully connected feed-forward network

   • Same structure as encoder's feed-forward network

---

Attention Mechanism

Scaled Dot-Product Attention

\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V

• Q (queries), K (keys), V (values) are matrices
• Scaling by \frac{1}{\sqrt{d_k}} prevents gradients from becoming too small
• Faster and more space-efficient than additive attention

---

Multi-Head Attention

• Projects queries, keys, values h times with different learned projections
• Performs attention in parallel on projected versions
• Concatenates results and projects again

\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, ..., \text{head}_h)W^O

---

Attention Application: Encoder-Decoder

Encoder-decoder attention:

• Queries come from previous decoder layer
• Memory keys and values come from encoder output
• Allows every position in decoder to attend over all positions in input sequence
• Mimics typical encoder-decoder attention mechanisms

---

Attention Application: Encoder Self-Attention

Encoder self-attention:

• Keys, values and queries all come from previous encoder layer
• Each position attends to all positions in previous encoder layer
• Enables modeling of relationships between all words in input sequence
• No regard to distance between positions

---

Attention Application: Decoder Self-Attention

Decoder self-attention:

• Keys, values and queries come from previous decoder layer
• Each position attends to all positions up to and including itself
• Masking prevents attending to subsequent positions
• Preserves auto-regressive property (predictions depend only on known outputs)

---

Positional Encoding

Since model has no recurrence/convolution, we inject positional information:

\text{PE}_{(pos, 2i)} = \sin\left(pos / 10000^{2i/d_{\text{model}}}\right) \text{PE}_{(pos, 2i+1)} = \cos\left(pos / 10000^{2i/d_{\text{model}}}\right)

• Same dimension as embeddings (d_{\text{model}})
• Allows model to learn relative position information
• Performed nearly as well as learned positional embeddings

---

Why Self-Attention?

Layer Type	Complexity	Sequential Operations	Max Path Length
Self-Attention	O(n^2 \cdot d)	O(1)	O(1)
Recurrent	O(n \cdot d^2)	O(n)	O(n)
Convolutional	O(k \cdot n \cdot d^2)	O(1)	O(\log_k n)

---

Training: Data & Batching

• WMT 2014 English-German (4.5M sentence pairs)
• WMT 2014 English-French (36M sentence pairs)
• Byte-pair encoding (37K shared vocab for EN-DE)
• Batches with ~25000 source and target tokens

---

Training: Hardware & Schedule

• 8 NVIDIA P100 GPUs
• Base model: 100,000 steps (12 hours)
• Big model: 300,000 steps (3.5 days)
• Adam optimizer with scheduled learning rate

---

Machine Translation Results

Model	EN-DE BLEU	EN-FR BLEU	Training Cost (FLOPs)
GNMT + RL Ensemble	26.30	41.16	1.8 \cdot 10^{20}
ConvS2S Ensemble	26.36	41.29	7.7 \cdot 10^{19}
Transformer (big)	28.4	41.8	$2.3 \cdot 10^{19}$

---

Generalization to Constituency Parsing

Parser	Training	WSJ 23 F1
Previous state-of-the-art	WSJ only	91.7
Transformer (4 layers)	WSJ only	91.3
Previous state-of-the-art	Semi-supervised	92.1
Transformer (4 layers)	Semi-supervised	92.7

---

Attention Visualization: Long-Distance Dependencies

---

Attention Visualization: Anaphora Resolution

---

Limitations

• Computational complexity grows quadratically with sequence length
• Less effective for very long sequences
• Still requires sequential generation in decoder
• Limited ability to model hierarchical structure

---

Key Contributions

• Introduced Transformer architecture based solely on attention
• Achieved new state-of-the-art results in machine translation
• Demonstrated improved parallelization and reduced training time
• Showed generalization to other tasks like constituency parsing

---

Future Work

• Apply to other modalities (images, audio, video)
• Investigate local, restricted attention for large inputs
• Make generation less sequential
• Explore interpretability of attention mechanisms

---

Thank You

Code available at: https://github.com/tensorflow/tensor2tensor

arXiv:1706.03762v7 [cs.CL]