04 · METHOD
Scaled Dot-Product & Multi-Head Attention
Attention(Q,K,V) = softmax(QK^T / √d_k)V
- Attention maps a query and key–value pairs to an output: a weighted sum of values, weights from query–key compatibility.
- Dot products are scaled by 1/√d_k — for large d_k they would push softmax into tiny-gradient regions.
- Multi-head: queries, keys, values are linearly projected h = 8 times (to d_k = d_v = 64), attention runs in parallel, outputs are concatenated and re-projected.
- Different heads jointly attend to information from different representation subspaces at different positions.
(left) Scaled Dot-Product Attention; (right) Multi-Head Attention — several attention layers in parallel. Figure 2, paper p. 4.
Visual: Figure 2 (paper p. 4); formula and text: §3.2 (paper p. 4–5).