11 · RESULTS
Interpretability: What Attention Sees
- Individual attention heads learn to perform different tasks; many exhibit behavior tied to the syntactic and semantic structure of sentences.
- In this example (encoder self-attention, layer 5 of 6), many heads attend to a distant dependency of the verb “making”, completing the phrase “making…more difficult”.
- Other heads are involved in anaphora resolution (Figure 4 of the paper).
“Not only do individual attention heads clearly learn to perform different tasks, many appear to exhibit behavior related to the syntactic and semantic structure of the sentences.” — §4, p. 7
Attention for the word “making”; colors = different heads. Figure 3, paper p. 13.
Visual: Figure 3 (paper p. 13); text: §4 (paper p. 7) and Appendix (paper p. 13).