Attention Is All You Need

This paper introduced the Transformer as an encoder–decoder sequence-transduction architecture. Its durable contribution is not merely “use attention”: it specifies how attention, position-wise feed-forward networks, residual paths, normalization, positional information, and masking compose into a trainable sequence model.

Mechanism #

Scaled dot-product attention maps queries, keys, and values through

$$\mathop{\mathrm{Attention}}(Q,K,V)=\mathop{\mathrm{softmax}}\left(\frac{QK^T}{\sqrt{d_k}}\right)V.$$

The $1/\sqrt{d_k}$ factor limits dot-product magnitude as key dimension grows, preventing softmax from becoming needlessly saturated. Multi-head attention performs this operation in several learned subspaces and concatenates the results.

The original model has two distinct stacks. The encoder uses unmasked self-attention over the source sequence. The decoder uses masked self-attention over the generated prefix, then cross-attends to encoder outputs. Setting disallowed attention logits to negative infinity makes the corresponding softmax weights zero; together with the shifted target sequence, this prevents a training position from reading the token it must predict.

Because attention itself contains no order, the paper adds sinusoidal or learned positional encodings to token embeddings. Each attention sublayer and position-wise feed-forward sublayer is wrapped by a residual connection and layer normalization.

Results worth retaining #

The paper's complexity comparison isolates why self-attention was attractive for fixed-length sequence transduction: all positions can be processed in parallel during training, and the maximum path between any two positions is constant in layer count. Dense attention pays $O(n^2d)$ work and materializes an $n\times n$ interaction pattern, so the same global connectivity becomes expensive as context length grows.

The translation scores and 2017 training hardware are historical evidence, not present-day model recommendations.

Scope and limitation #

The diagram most often reproduced from this paper is an encoder–decoder Transformer. It is not a diagram of a GPT-style decoder-only language model. A decoder-only model keeps the masked self-attention stack and next-token objective but has no separate source encoder and no encoder–decoder cross-attention sublayer.

Visual audit #

All 11 physical pages were rendered and inspected. Figures 1 and 2 are information-bearing: they show the original encoder–decoder topology and scaled dot-product/multi-head attention. They are not retained because Figure 1 would misstate the architecture being taught in the GTK GPT chapter, while the attention formula and head composition are already explained in the architecture entry. GTK uses an independently drawn causal-decoder flow instead.