This comprehensive 7-hour freeCodeCamp course takes you through the complete evolution of neural machine translation, from basic RNNs to modern Transformers.
Course Overview
The journey covers:
- Recurrent Neural Networks (RNNs)
- Long Short-Term Memory (LSTM)
- Sequence-to-Sequence Models
- Attention Mechanism
- Transformer Architecture
Part 1: Recurrent Neural Networks
Why RNNs for Sequences?
- Traditional neural networks can’t handle variable-length sequences
- RNNs process one element at a time
- Maintain hidden state across time steps
RNN Architecture
h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b)
y_t = W_hy * h_t + b_y
- h_t: Hidden state at time t
- x_t: Input at time t
- W: Weight matrices
The Vanishing Gradient Problem
- Gradients shrink exponentially over long sequences
- Early time steps receive tiny gradients
- Network “forgets” long-range dependencies
Part 2: LSTM Networks
Solving Vanishing Gradients
LSTM introduces gates to control information flow:
Forget Gate:
f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
Input Gate:
i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
Output Gate:
o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
Cell State:
C_t = f_t * C_{t-1} + i_t * tanh(W_C · [h_{t-1}, x_t] + b_C)
Why LSTM Works
- Cell state provides “highway” for gradients
- Gates learn what to remember/forget
- Can capture long-range dependencies
GRU: Simplified Alternative
- Combines forget and input gates
- Fewer parameters than LSTM
- Often similar performance
Part 3: Sequence-to-Sequence
Encoder-Decoder Architecture
[Input Sequence] → Encoder → [Context Vector] → Decoder → [Output Sequence]
Encoder:
- Processes input sequence
- Produces fixed-size context vector
- Captures input meaning
Decoder:
- Takes context vector
- Generates output sequence
- One token at a time
The Bottleneck Problem
- Entire input compressed into single vector
- Information loss for long sequences
- Decoder has limited access to input
Part 4: Attention Mechanism
The Key Insight
Instead of single context vector, let decoder “attend” to all encoder states.
Attention Calculation
- Score: How relevant is each encoder state?
- Weights: Softmax over scores
- Context: Weighted sum of encoder states
scores = dot(decoder_state, encoder_states)
weights = softmax(scores)
context = sum(weights * encoder_states)
Types of Attention
- Dot-product attention: Simple, fast
- Additive attention: More parameters, sometimes better
- Multi-head attention: Multiple attention patterns
Benefits
- No information bottleneck
- Decoder can focus on relevant parts
- Interpretable (attention weights show focus)
Part 5: Transformer Architecture
”Attention Is All You Need”
The 2017 paper that changed NLP:
- No recurrence needed
- Parallel processing
- Better long-range dependencies
Self-Attention
Each position attends to all positions:
Attention(Q, K, V) = softmax(QK^T / √d_k) V
Multi-Head Attention
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) W^O
where head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)
Transformer Block
- Multi-head self-attention
- Add & Layer Norm
- Feed-forward network
- Add & Layer Norm
Positional Encoding
Since no recurrence, add position information:
PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Encoder-Decoder in Transformer
- Encoder: Self-attention only
- Decoder: Self-attention + cross-attention to encoder
Implementation Highlights
Training Tips
- Label smoothing
- Learning rate warmup
- Dropout for regularization
- Gradient clipping
Inference
- Beam search for better outputs
- Length normalization
- Early stopping
Evolution Summary
| Model | Year | Key Innovation |
|---|---|---|
| RNN | 1986 | Sequential processing |
| LSTM | 1997 | Gated memory cells |
| Seq2Seq | 2014 | Encoder-decoder |
| Attention | 2015 | Dynamic context |
| Transformer | 2017 | Self-attention, parallelism |
Modern Applications
- Machine translation (Google Translate)
- Text summarization
- Question answering
- Code generation
- Foundation for GPT, BERT, etc.
这门全面的7小时freeCodeCamp课程带你了解神经机器翻译的完整演进，从基础RNN到现代Transformer。
课程概述
旅程涵盖：
- 循环神经网络（RNN）
- 长短期记忆（LSTM）
- 序列到序列模型
- 注意力机制
- Transformer架构
第一部分：循环神经网络
为什么用RNN处理序列？
- 传统神经网络无法处理可变长度序列
- RNN一次处理一个元素
- 跨时间步维护隐藏状态
RNN架构
h_t = tanh(W_hh * h_{t-1} + W_xh * x_t + b)
y_t = W_hy * h_t + b_y
- h_t：时间t的隐藏状态
- x_t：时间t的输入
- W：权重矩阵
梯度消失问题
- 梯度在长序列上指数级缩小
- 早期时间步收到微小梯度
- 网络”遗忘”长距离依赖
第二部分：LSTM网络
解决梯度消失
LSTM引入门控制信息流：
遗忘门：
f_t = σ(W_f · [h_{t-1}, x_t] + b_f)
输入门：
i_t = σ(W_i · [h_{t-1}, x_t] + b_i)
输出门：
o_t = σ(W_o · [h_{t-1}, x_t] + b_o)
细胞状态：
C_t = f_t * C_{t-1} + i_t * tanh(W_C · [h_{t-1}, x_t] + b_C)
为什么LSTM有效
- 细胞状态为梯度提供”高速公路”
- 门学习记住/遗忘什么
- 能捕获长距离依赖
GRU：简化替代
- 合并遗忘门和输入门
- 比LSTM参数更少
- 通常性能相似
第三部分：序列到序列
编码器-解码器架构
[输入序列] → 编码器 → [上下文向量] → 解码器 → [输出序列]
编码器：
- 处理输入序列
- 产生固定大小上下文向量
- 捕获输入含义
解码器：
- 接收上下文向量
- 生成输出序列
- 一次一个token
瓶颈问题
- 整个输入压缩成单个向量
- 长序列信息丢失
- 解码器对输入访问有限
第四部分：注意力机制
关键洞察
不用单个上下文向量，让解码器”关注”所有编码器状态。
注意力计算
- 分数：每个编码器状态有多相关？
- 权重：分数的softmax
- 上下文：编码器状态的加权和
scores = dot(decoder_state, encoder_states)
weights = softmax(scores)
context = sum(weights * encoder_states)
注意力类型
- 点积注意力：简单、快速
- 加性注意力：更多参数，有时更好
- 多头注意力：多种注意力模式
优势
- 无信息瓶颈
- 解码器可关注相关部分
- 可解释（注意力权重显示焦点）
第五部分：Transformer架构
”注意力就是你所需要的”
2017年改变NLP的论文：
- 不需要循环
- 并行处理
- 更好的长距离依赖
自注意力
每个位置关注所有位置：
Attention(Q, K, V) = softmax(QK^T / √d_k) V
多头注意力
MultiHead(Q, K, V) = Concat(head_1, ..., head_h) W^O
其中 head_i = Attention(QW_i^Q, KW_i^K, VW_i^V)
Transformer块
- 多头自注意力
- 加法和层归一化
- 前馈网络
- 加法和层归一化
位置编码
由于没有循环，添加位置信息：
PE(pos, 2i) = sin(pos / 10000^(2i/d))
PE(pos, 2i+1) = cos(pos / 10000^(2i/d))
Transformer中的编码器-解码器
- 编码器：仅自注意力
- 解码器：自注意力 + 对编码器的交叉注意力
实现要点
训练技巧
- 标签平滑
- 学习率预热
- Dropout正则化
- 梯度裁剪
推理
- 束搜索获得更好输出
- 长度归一化
- 提前停止
演进总结
| 模型 | 年份 | 关键创新 |
|---|---|---|
| RNN | 1986 | 序列处理 |
| LSTM | 1997 | 门控记忆单元 |
| Seq2Seq | 2014 | 编码器-解码器 |
| 注意力 | 2015 | 动态上下文 |
| Transformer | 2017 | 自注意力、并行 |
现代应用
- 机器翻译（Google翻译）
- 文本摘要
- 问答
- 代码生成
- GPT、BERT等的基础