English

Overview

The paper “Outrageously Large Neural Networks： The Sparsely-Gated Mixture-of-Experts Layer” introduces a revolutionary approach to scaling neural networks. By using a sparsely-gated mixture of experts, models can scale to billions of parameters while maintaining constant computational cost per example.

Key Contributions

  1. Sparse Gating: Only a subset of experts are activated for each input, dramatically reducing computation
  2. Conditional Computation: Different parts of the network are used for different inputs
  3. Massive Scale: Demonstrated models with up to 137 billion parameters
  4. Efficiency: Achieved better results than dense models with similar computational budgets

Architecture

The MoE layer consists of：

  • Expert Networks: Multiple feed-forward networks (experts)
  • Gating Network: Learns which experts to activate for each input
  • Sparse Activation: Only top-k experts are used per input

Impact

MoE has become foundational for modern large language models：

  • GPT-4 is rumored to use MoE architecture
  • Mixtral and other open models use MoE
  • Enables training much larger models efficiently

arXiv

.06538

中文

概述

论文《超大规模神经网络：稀疏门控专家混合层》介绍了一种革命性的神经网络扩展方法。通过使用稀疏门控的专家混合，模型可以扩展到数十亿参数，同时保持每个样本的计算成本恒定。

主要贡献

  1. 稀疏门控：每个输入只激活部分专家，大幅减少计算量
  2. 条件计算：网络的不同部分用于不同的输入
  3. 大规模：展示了高达1370亿参数的模型
  4. 高效性：在相似计算预算下取得了比密集模型更好的结果

架构

MoE层包含：

  • 专家网络：多个前馈网络（专家）
  • 门控网络：学习为每个输入激活哪些专家
  • 稀疏激活：每个输入只使用top-k个专家

影响

MoE已成为现代大型语言模型的基础：

  • GPT-4据传使用MoE架构
  • Mixtral等开源模型使用MoE
  • 使得高效训练更大模型成为可能

论文链接

arXiv

.06538