Attention Is All You Need: The Paper That Started the AI Revolution

Paper: 1706.03762 Authors: Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin Venue: NeurIPS 2017 Categories: cs.CL, cs.LG

Abstract

The dominant sequence transduction models are based on complex recurrent or convolutional neural networks that include an encoder and a decoder. The best performing models also connect the encoder and decoder through an attention mechanism. We propose a new simple network architecture, the Transformer, based solely on attention mechanisms, dispensing with recurrence and convolutions entirely.

Key Contributions

• The Transformer Architecture: A novel neural network that relies entirely on self-attention, eliminating the need for recurrence and convolutions
• Scaled Dot-Product Attention: Efficient attention mechanism with scaling factor to prevent gradient issues
• Multi-Head Attention: Allows the model to attend to information from different representation subspaces at different positions
• Positional Encoding: Sinusoidal functions to inject sequence order information without recurrence

The Problem with RNNs

Before Transformers, sequence models relied on RNNs (LSTM, GRU):

• Sequential computation: Cannot parallelize within training examples
• Long-range dependencies: Difficult to learn due to vanishing gradients
• Memory constraints: Limit batching across examples at longer sequences

Transformer Architecture

Encoder-Decoder Structure

Encoder: 6 identical layers
  - Multi-head self-attention
  - Position-wise feed-forward network
  - Residual connections + Layer normalization

Decoder: 6 identical layers
  - Masked multi-head self-attention
  - Multi-head attention over encoder output
  - Position-wise feed-forward network

Scaled Dot-Product Attention

The core attention formula:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V$

Why scale by $\sqrt{d_k}$? For large $d_k$, dot products grow large, pushing softmax into regions with extremely small gradients.

Multi-Head Attention

Instead of single attention:

$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \ldots, \text{head}_h) W^O$

where each head is:

$\text{head}_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)$

With $h=8$ heads, $d_k = d_v = d_{model}/h = 64$

Position-wise Feed-Forward Networks

Applied to each position separately:

$\text{FFN}(x) = \max(0, xW_1 + b_1)W_2 + b_2$

Inner layer dimension: 2048, outer: 512

Positional Encoding

Since no recurrence, position info added via:

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

Results

Machine Translation

Model	EN-DE BLEU	EN-FR BLEU	Training Cost
Previous SOTA	26.4	41.0	-
Transformer (base)	27.3	38.1	12 hours, 8 GPUs
Transformer (big)	28.4	41.8	3.5 days, 8 GPUs

Why Self-Attention?

Compared to recurrent and convolutional layers:

Layer Type	Complexity per Layer	Sequential Ops	Max Path Length
Self-Attention	$O(n^2 \cdot d)$	$O(1)$	$O(1)$
Recurrent	$O(n \cdot d^2)$	$O(n)$	$O(n)$
Convolutional	$O(k \cdot n \cdot d^2)$	$O(1)$	$O(\log_k(n))$

Self-attention connects all positions with $O(1)$ operations.

Takeaways

1. Attention alone is sufficient - No need for recurrence or convolution for sequence transduction
2. Parallelization wins - Training is dramatically faster than RNN-based models
3. Constant path length - Any two positions can interact directly, improving long-range dependency learning
4. Scalability - The architecture scales well with more data and compute

This paper fundamentally changed NLP and later computer vision, leading to BERT, GPT, and all modern large language models.

编码器：6个相同的层
  - 多头自注意力
  - 逐位置前馈网络
  - 残差连接 + 层归一化

解码器：6个相同的层
  - 掩码多头自注意力
  - 对编码器输出的多头注意力
  - 逐位置前馈网络