论文信息
- 标题: Distilling the Knowledge in a Neural Network
- 作者: Geoffrey Hinton, Oriol Vinyals, Jeff Dean
- 发表年份: 2015
- arXiv: 1503.02531
核心贡献
本文提出了知识蒸馏（Knowledge Distillation）方法，通过使用软目标（soft targets）和温度参数（temperature），将大型复杂模型（教师模型）的知识迁移到小型高效模型（学生模型）中。
主要内容
软目标与温度
传统训练使用硬标签（one-hot），而知识蒸馏使用教师模型的软概率分布作为目标。通过提高温度参数T，可以软化概率分布，揭示类别间的相似性关系。
蒸馏损失
学生模型的训练结合两个损失：
- 与教师软目标的交叉熵（高温）
- 与真实标签的交叉熵（正常温度）
模型压缩
实验表明，小型学生模型可以达到接近大型教师模型的性能，同时推理速度更快、资源消耗更少。
影响与应用
知识蒸馏已成为模型压缩的标准技术，广泛应用于：
- 移动端模型部署
- 边缘设备推理
- 模型集成的知识提取
- BERT等大模型的压缩（DistilBERT）
Paper Information
- Title: Distilling the Knowledge in a Neural Network
- Authors: Geoffrey Hinton, Oriol Vinyals, Jeff Dean
- Year: 2015
- arXiv: 1503.02531
Core Contribution
This paper proposes Knowledge Distillation, a method to transfer knowledge from large complex models (teacher) to small efficient models (student) using soft targets and temperature scaling.
Main Content
Soft Targets and Temperature
Traditional training uses hard labels (one-hot), while knowledge distillation uses the teacher model’s soft probability distribution as targets. By raising the temperature parameter T, the probability distribution is softened, revealing similarity relationships between classes.
Distillation Loss
Student model training combines two losses：
- Cross-entropy with teacher’s soft targets (high temperature)
- Cross-entropy with true labels (normal temperature)
Model Compression
Experiments show that small student models can achieve performance close to large teacher models while being faster and more resource-efficient.
Impact and Applications
Knowledge distillation has become a standard technique for model compression, widely used in：
- Mobile model deployment
- Edge device inference
- Knowledge extraction from model ensembles
- Compression of large models like BERT (DistilBERT)