03 · METHOD

Model Architecture: Stacked Self-Attention

• Encoder–decoder structure built from stacked self-attention and point-wise, fully connected layers (left and right halves of Figure 1).
• Encoder: N = 6 identical layers — each with multi-head self-attention, then a position-wise feed-forward network.
• Decoder: N = 6 layers; adds a masked multi-head self-attention sub-layer (positions only attend to earlier positions) plus attention over the encoder output.
• A residual connection wraps every sub-layer, followed by layer normalization.
• Inputs/outputs are embedded (d_model = 512) and combined with positional encodings, since the model has no notion of order on its own.

Encoder (left) and decoder (right) stacks.
Figure 1, paper p. 3.

Visual: Figure 1, “The Transformer - model architecture” (paper p. 3); text: §3.1 (paper p. 3).