Paper: arXiv
.14165 Authors: Tom Brown, Benjamin Mann, Nick Ryder, et al. (OpenAI) Venue: NeurIPS 2020Core Contributions
GPT-3 demonstrated the power of scale: a sufficiently large language model can complete various tasks through simple text prompts without task-specific fine-tuning.
Model Scale
| Model | Parameters | Layers | Hidden Dim | Attention Heads |
|---|---|---|---|---|
| GPT-3 Small | 125M | 12 | 768 | 12 |
| GPT-3 Medium | 350M | 24 | 1024 | 16 |
| GPT-3 Large | 760M | 24 | 1536 | 16 |
| GPT-3 XL | 1.3B | 24 | 2048 | 24 |
| GPT-3 6.7B | 6.7B | 32 | 4096 | 32 |
| GPT-3 175B | 175B | 96 | 12288 | 96 |
Key Innovations
In-Context Learning
GPT-3 introduced three learning paradigms:
Zero-shot: Only task description
Translate English to French:
cheese =>
One-shot: One example
Translate English to French:
sea otter => loutre de mer
cheese =>
Few-shot: Several examples
Translate English to French:
sea otter => loutre de mer
peppermint => menthe poivrée
cheese =>
Emergent Abilities
As model scale increases, GPT-3 exhibits emergent abilities not present in smaller models:
- Arithmetic: 100% accuracy on two-digit addition/subtraction
- Word unscrambling: Rearranging scrambled words into sentences
- Novel word definition: Understanding new words from context
Experimental Results
Few-shot GPT-3 approaches or exceeds fine-tuned BERT on multiple tasks:
| Task | BERT (fine-tuned) | GPT-3 (Few-shot) |
|---|---|---|
| SuperGLUE | 71.5 | 71.8 |
| TriviaQA | 68.0 | 71.2 |
| Translation (En→Fr) | 40.2 BLEU | 32.6 BLEU |
Limitations
The paper honestly discusses GPT-3’s limitations:
- Text generation: Long texts may lose coherence
- Common sense reasoning: Poor performance on some physical common sense questions
- Bidirectional tasks: Fill-in-the-blank tasks worse than BERT
- Sample efficiency: Requires massive computational resources
Lasting Impact
GPT-3 launched the Large Language Model (LLM) era:
- Proved the effectiveness of Scaling Laws
- Spawned the Prompt Engineering field
- Drove development of ChatGPT, Claude, and other conversational AI
- Sparked widespread discussion about AI safety and alignment
GPT-3 is a milestone in AI development, marking the paradigm shift from “training models for tasks” to “guiding general models with prompts.”
论文: arXiv
.14165 作者: Tom Brown, Benjamin Mann, Nick Ryder 等 (OpenAI) 会议: NeurIPS 2020核心贡献
GPT-3 证明了规模化的力量：一个足够大的语言模型可以通过简单的文本提示（prompt）完成各种任务，无需针对特定任务进行微调。
模型规模
| 模型 | 参数量 | 层数 | 隐藏维度 | 注意力头 |
|---|---|---|---|---|
| GPT-3 Small | 125M | 12 | 768 | 12 |
| GPT-3 Medium | 350M | 24 | 1024 | 16 |
| GPT-3 Large | 760M | 24 | 1536 | 16 |
| GPT-3 XL | 1.3B | 24 | 2048 | 24 |
| GPT-3 6.7B | 6.7B | 32 | 4096 | 32 |
| GPT-3 175B | 175B | 96 | 12288 | 96 |
关键创新
上下文学习
GPT-3 引入了三种学习范式：
Zero-shot: 只给任务描述
Translate English to French：
cheese =>
One-shot: 给一个示例
Translate English to French：
sea otter => loutre de mer
cheese =>
Few-shot: 给几个示例
Translate English to French：
sea otter => loutre de mer
peppermint => menthe poivrée
cheese =>
涌现能力
随着模型规模增大，GPT-3 展现出一些小模型没有的能力：
- 算术运算: 两位数加减法准确率达 100%
- 单词重排: 将打乱的单词重新排列成句子
- 新词定义: 根据上下文理解新造词的含义
实验结果
Few-shot GPT-3 在多个任务上接近或超过微调的 BERT：
| 任务 | BERT (微调) | GPT-3 (Few-shot) |
|---|---|---|
| SuperGLUE | 71.5 | 71.8 |
| TriviaQA | 68.0 | 71.2 |
| 翻译 (En→Fr) | 40.2 BLEU | 32.6 BLEU |
局限性
论文诚实地讨论了 GPT-3 的局限：
- 文本生成: 长文本可能失去连贯性
- 常识推理: 某些物理常识问题表现不佳
- 双向任务: 填空类任务不如 BERT
- 样本效率: 需要大量计算资源
深远影响
GPT-3 开启了大语言模型 (LLM) 时代：
- 证明了 Scaling Laws 的有效性
- 催生了 Prompt Engineering 领域
- 推动了 ChatGPT、Claude 等对话 AI 的发展
- 引发了关于 AI 安全和对齐的广泛讨论
GPT-3 是 AI 发展的里程碑，标志着从”针对任务训练模型”到”用提示引导通用模型”的范式转变。