Chinchilla: Training Compute-Optimal Large Language Models / Chinchilla: Training Compute-Optimal Large Language Models

论文信息

标题: Training Compute-Optimal Large Language Models
作者: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al.
机构: DeepMind
发表时间: 2022年3月
论文链接: arXiv
.15556

核心贡献

Chinchilla 论文挑战了”越大越好”的观点，发现当时的大模型（如 Gopher、GPT-3）都严重欠训练。在相同计算预算下，更小但训练更充分的模型可以获得更好的性能。

主要发现

最优比例: 模型参数和训练 tokens 应该等比例增加
Chinchilla 法则: 每个参数应该对应约 20 个训练 tokens
Gopher 欠训练: 280B 参数的 Gopher 应该用 5.6T tokens 训练，而非 300B
效率提升: 70B 参数的 Chinchilla 超越了 280B 的 Gopher

核心公式

最优模型大小

给定计算预算 C，最优模型大小：

$N_{opt} \propto C^{0.5}$

最优训练 tokens

$D_{opt} \propto C^{0.5}$

Chinchilla 法则

$D_{opt} \approx 20 \times N$

即每个参数需要约 20 个训练 tokens。

实验方法

论文使用三种方法验证结论：

固定模型大小，变化训练量: 训练不同大小模型到不同 tokens
IsoFLOP 曲线: 固定计算量，变化模型大小
拟合参数化损失函数: 建立损失与 N、D 的函数关系

Chinchilla vs Gopher

模型	参数量	训练 Tokens	计算量	性能
Gopher	280B	300B	5.76×10²³	基准
Chinchilla	70B	1.4T	5.76×10²³	更优

关键洞察：相同计算预算下，Chinchilla 用 4 倍数据训练 1/4 大小的模型，性能更好。

对行业的影响

1. 改变训练策略

从追求更大模型转向更充分训练
LLaMA 等后续模型采用 Chinchilla 法则

2. 数据重要性

强调了高质量训练数据的重要性
推动了数据收集和清洗工作

3. 推理效率

更小的模型意味着更低的推理成本
有利于模型部署和应用

与 OpenAI Scaling Laws 的对比

方面	OpenAI (2020)	Chinchilla (2022)
最优 N/D 比例	N 增长更快	N 和 D 等比例增长
数据需求	相对较少	更多数据
模型大小偏好	更大模型	适中模型

总结

Chinchilla 论文是 scaling laws 研究的重要修正，它表明在计算预算有限的情况下，应该平衡模型大小和训练数据量，而非一味追求更大的模型。这一发现深刻影响了后续 LLM 的设计和训练策略。

Paper Information

Title: Training Compute-Optimal Large Language Models
Authors: Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, et al.
Institution: DeepMind
Published: March 2022
Paper Link: arXiv
.15556

Core Contributions

The Chinchilla paper challenged the “bigger is better” view, finding that large models at the time (like Gopher, GPT-3) were severely undertrained. With the same compute budget, smaller but more thoroughly trained models can achieve better performance.

Key Findings

Optimal Ratio: Model parameters and training tokens should increase proportionally
Chinchilla Law: Each parameter should correspond to about 20 training tokens
Gopher Undertrained: 280B parameter Gopher should be trained on 5.6T tokens, not 300B
Efficiency Gains: 70B parameter Chinchilla outperforms 280B Gopher

The Chinchilla Law

The key insight is that optimal training requires approximately 20 tokens per parameter. This means a 70B model should be trained on 1.4T tokens.

Industry Impact

1. Changed Training Strategy

Shifted from pursuing larger models to more thorough training
Subsequent models like LLaMA adopted Chinchilla law

2. Data Importance

Emphasized the importance of high-quality training data
Drove data collection and cleaning efforts

3. Inference Efficiency

Smaller models mean lower inference costs
Benefits model deployment and applications

Summary

The Chinchilla paper is an important correction to scaling laws research, showing that with limited compute budget, one should balance model size and training data rather than blindly pursuing larger models. This finding profoundly influenced subsequent LLM design and training strategies.