01 · BACKGROUND

The Problem: The Limits of Recurrence

Sequence transduction (e.g., machine translation) was dominated by recurrent (LSTM / GRU) and convolutional encoder–decoder models.
RNNs compute hidden states one position at a time: inherently sequential, so training cannot parallelize across positions in a sequence.
Signals between distant positions must travel a path of O(n) operations — the longer the path, the harder long-range dependencies are to learn.
Attention mechanisms already existed, but only as an accessory layered on top of RNNs or CNNs.

O(n)

sequential operations per layer in a recurrent network

O(n)

maximum path length between any two positions

Source: Vaswani et al., “Attention Is All You Need”, §1 Introduction (paper p. 1–2).