论文信息

  • 标题: Scaling Laws for Neural Language Models
  • 作者: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al.
  • 机构: OpenAI, Johns Hopkins University
  • 发表时间: 2020年1月
  • 论文链接: arXiv
    .08361

核心贡献

这篇论文系统研究了语言模型性能与三个关键因素的关系：模型参数量 (N)、数据集大小 (D) 和计算量 (C)。发现它们之间存在简洁的幂律关系。

主要发现

  1. 幂律关系: 损失与 N、D、C 呈幂律下降
  2. 规模优先: 更大的模型更加样本高效
  3. 最优分配: 给定计算预算，存在最优的模型大小和数据量分配
  4. 可预测性: 可以从小规模实验预测大规模性能

核心公式

损失与参数量

L(N)≈(NcN)αNL(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}

其中 αN≈0.076\alpha_N \approx 0.076，Nc≈8.8×1013N_c \approx 8.8 \times 10^{13}

损失与数据量

L(D)≈(DcD)αDL(D) \approx \left(\frac{D_c}{D}\right)^{\alpha_D}

其中 αD≈0.095\alpha_D \approx 0.095，Dc≈5.4×1013D_c \approx 5.4 \times 10^{13}

损失与计算量

L(C)≈(CcC)αCL(C) \approx \left(\frac{C_c}{C}\right)^{\alpha_C}

其中 αC≈0.050\alpha_C \approx 0.050

关键洞察

1. 模型规模最重要

在固定计算预算下，应该优先增加模型大小而非训练更长时间。

2. 数据效率

大模型比小模型更加数据高效——达到相同性能需要更少的数据。

3. 最优计算分配

给定计算预算 C，最优策略是：

  • 模型大小 N∝C0.73N \propto C^{0.73}
  • 数据量 D∝C0.27D \propto C^{0.27}

4. 收敛前停止

最优策略是在模型完全收敛之前就停止训练。

实验设置

  • 模型大小： 768 到 15 亿参数
  • 数据集： WebText2
  • 架构： Transformer decoder-only
  • 训练： 从头训练，无预训练

对后续研究的影响

这篇论文深刻影响了 AI 研究方向：

  1. GPT-3: 基于 scaling laws 设计了 175B 参数模型
  2. Chinchilla: 重新审视了最优数据/模型比例
  3. 大模型竞赛: 推动了各公司追求更大模型
  4. 计算规划: 帮助研究者规划训练资源

局限性

  • 主要针对 Transformer 语言模型
  • 未考虑推理成本
  • 后续 Chinchilla 论文修正了部分结论

总结

Scaling Laws 论文揭示了语言模型性能的可预测性，为大规模模型训练提供了理论指导，是理解现代 LLM 发展的关键文献。

Paper Information

  • Title: Scaling Laws for Neural Language Models
  • Authors: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, et al.
  • Institution: OpenAI, Johns Hopkins University
  • Published: January 2020
  • Paper Link: arXiv
    .08361

Core Contributions

This paper systematically studies the relationship between language model performance and three key factors： model parameters (N), dataset size (D), and compute (C). It discovers simple power-law relationships between them.

Key Findings

  1. Power Laws: Loss decreases as a power law with N, D, C
  2. Scale Priority: Larger models are more sample efficient
  3. Optimal Allocation: Given compute budget, there’s optimal model size and data allocation
  4. Predictability: Large-scale performance can be predicted from small-scale experiments

Core Formulas

The paper establishes that loss follows power-law relationships with model size, data, and compute, with specific exponents that allow prediction of model performance.

Key Insights

1. Model Size Matters Most

With fixed compute budget, prioritize increasing model size over training longer.

2. Data Efficiency

Large models are more data efficient than small models—they need less data to achieve the same performance.

3. Optimal Compute Allocation

Given compute budget C, optimal strategy allocates more to model size than data.

4. Stop Before Convergence

Optimal strategy is to stop training before the model fully converges.

Impact on Subsequent Research

This paper profoundly influenced AI research：

  1. GPT-3: Designed 175B parameter model based on scaling laws
  2. Chinchilla: Re-examined optimal data/model ratio
  3. Large Model Race: Drove companies to pursue larger models
  4. Compute Planning: Helped researchers plan training resources

Summary

The Scaling Laws paper revealed the predictability of language model performance, providing theoretical guidance for large-scale model training and serving as key literature for understanding modern LLM development.