Parallelization: a self-attention layer needs only O(1) sequential operations vs. O(n) for recurrence — training uses all positions at once.
Long-range learning: any two positions connect through a path of constant length O(1), instead of O(n) for RNNs or O(log_k n) for dilated convolutions.
Cost: per-layer complexity O(n²·d) beats recurrence O(n·d²) whenever n < d — the common case with word-piece / BPE representations.
Interpretability: attention distributions expose what the model attends to; heads learn distinct, task-like behaviors.
PER-LAYER COMPLEXITY AND PATH LENGTHS
n = sequence length, d = representation dimension, k = conv. kernel width, r = neighborhood. Adapted from Table 1, paper p. 6.
Table: adapted from Table 1 (paper p. 6); text: §4 (paper p. 6–7).