02 · KEY IDEA

The Transformer: Attention Is All You Need

No Recurrence

Sequential RNN hidden states are removed entirely — positions are processed in parallel.

No Convolution

No fixed-width kernels; every position can connect to every other position directly.

Self-Attention Only

Multi-headed self-attention computes all representations, in both encoder and decoder.

First sequence transduction model based entirely on attention, replacing recurrent layers with multi-headed self-attention.
Highly parallelizable — reaches a new state of the art in translation with a fraction of the training cost.
Generalizes beyond translation (e.g., English constituency parsing) and yields interpretable attention patterns.

Source: Vaswani et al., Abstract and §1 (paper p. 1–2).