05 · METHOD
Multi-Head Attention
h = 8 parallel heads; d_k = d_v = d_model/h = 64, so total compute stays close to single-head attention at full dimension.
Position-wise Feed-Forward
FFN(x) = max(0, xW₁+b₁)W₂+b₂, applied identically to every position; inner dimension d_ff = 2048.
Add & Norm
Residual connection around each sub-layer: LayerNorm(x + Sublayer(x)); all sub-layers and embeddings output d_model = 512.
Positional Encoding
Sine/cosine waves of different frequencies injected at the inputs, so the model can use token order; learned alternatives worked equally well.
Two configurations: base — d_model 512, d_ff 2048, h 8, dropout 0.1 | big — d_model 1024, d_ff 4096, h 16, dropout 0.3.
Source: §3.2–3.5 (paper p. 4–5); §5.4 and Table 3 footnotes (paper p. 7–8).