Multi-View MoE: Letting Every Router Shape Expert Computation

Paper: 2610.40577
Authors: Zelin Zhao, Tianchi Ma, Junyuan Shang, Shuohuan Wang, Dianhai Yu (ERNIE Team, Baidu)
Categories: cs.LG, cs.AI

The Gap

Sparse Mixture-of-Experts (MoE) architectures (pioneered by Switch Transformers, Mixtral, and DeepSeek-V2/V3) scale parameter capacity while keeping per-token inference FLOPs constant by routing each token to a top-$k$ subset of experts.

However, standard MoE models operate under a persistent, unaddressed structural limitation: the router-utility disconnect.

1. Weak Correlation in Deep Layers:

   • In conventional architectures, a single linear routing head computes routing logits and selects experts via top-$k$ gating.
   • However, dense empirical evaluation reveals that the Spearman rank correlation between single-router scores and true empirical expert utility ($L_2$ activation norm) drops drastically throughout the network, falling below $\rho < 0.05$ in middle and deep layers.
   • Because unselected experts receive no direct gradient feedback from that token, poor routing decisions become self-reinforcing traps.

2. The Dilemma of Enriching Expert Signals:

   • Prior attempts to improve routing fall into costly computational traps:
     • Activation-based routing (e.g., AoE) requires preliminary forward passes through all experts to measure intermediate activations before selection, inflating FLOPs.
     • Multi-representation approaches (e.g., token slicing or duplicate branches) process multiple hidden vectors through the same expert, multiplying expert computation and all-to-all communication volume across tensor-parallel ranks.

                    THE TRADITIONAL MOE BOTTLENECK

  Hidden State h
        │
   [Single Router] ──> Spearman rho < 0.05 with true expert utility in deep layers!
        │
   Top-k Selection
        │
        ├──> Expert E_1(h)  ──┐
        ├──> Expert E_2(h)  ──┼──> Weighted Sum ──> Output y
        └──> Expert E_k(h)  ──┘
        
  * PROBLEM: All selected experts receive the EXACT SAME input vector h,
    ignoring why each expert was chosen by the routing mechanism!

---

The Increment

One sentence: Multi-View MoE aggregates independently initialized sigmoid routers to rectify expert ranking, and reuses their relative preferences to mix a basis of shared latent views into expert-tailored inputs—achieving higher downstream accuracy and lower training loss under a strictly fixed budget of active experts, FLOPs, and communication.

                              MULTI-VIEW MOE ARCHITECTURE

                     Hidden State h (Dimension d)
                    ┌──────────────┴──────────────┐
                    │                             │
                    ▼                             ▼
        [Shared Projection Basis]         [Multi-Router Submodule]
       P_1 h, P_2 h, ..., P_K h           R independent router heads
           (K latent views)               s_{m,i} = sigmoid(w_m,i^T h)
                    │                             │
                    │               ┌─────────────┴─────────────┐
                    │               ▼                           ▼
                    │       Sum Across Routers:         Normalize Across
                    │        s_bar_i = sum_m s_m,i       Routers (per expert):
                    │               │                   a_m,i = s_m,i / sum_n s_n,i
                    │               ▼                           │
                    │       Top-k Selection S                   │
                    │               │                           │
                    └───────────────┼───────────────────────────┘
                                    │
                                    ▼
                         [Latent View Mixing]
             For each selected expert i in S:
             v_hat_i = sum_{m=1}^K a_m,i * (P_m h)
                                    │
                                    ▼
                         [Expert Forward Pass]
             Exactly ONE forward pass per active expert: E_i(v_hat_i)
                                    │
                                    ▼
                         [Routed Output Gating]
                     y = sum_{i in S} g_i * E_i(v_hat_i)

---

The Anatomy

1. Multi-Router: Aligning Routing with Utility

Instead of relying on a single fallible gating vector, Multi-Router instantiates $R$ independently parameterized routing heads $\{w^{(m)}\}_{m=1}^R$. For hidden state $h$, router $m$ outputs a sigmoid score for expert $i$: $s_{m,i} = \sigma\left({w^{(m)}_i}^\top h\right)$

The aggregate score for expert $i$ is obtained by summing across routers and adding the load-balancing bias $b_i$: $\bar{s}_i = \sum_{m=1}^R s_{m,i}$ $\mathcal{S} = \text{TopK}_k(\bar{s} + b)$

During training, these independent routers naturally differentiate: one router learns fine-grained syntactic boundaries while another tracks semantic reasoning roles. An expert can be selected if it is strongly favored by one specialist router or jointly endorsed by consensus across multiple routers.

2. Multi-View Input Modulation

Standard MoE feeds the identical projection $P h$ to every active expert. Multi-View MoE introduces a shared bank of $K$ linear projection matrices $\{P_m\}_{m=1}^K$, generating $K$ distinct latent views: $v^{(m)} = P_m h \in \mathbb{R}^{d_{\text{latent}}}$

For each selected expert $i \in \mathcal{S}$, Multi-View computes convex mixing coefficients by normalizing expert $i$‘s scores across the $R$ routers (where $R = K$): $a_{m,i} = \frac{s_{m,i}}{\sum_{n=1}^R s_{n,i}}, \quad \sum_{m=1}^K a_{m,i} = 1$

The input $\hat{v}_i$ delivered to expert $E_i$ is the convex combination of the latent views: $\hat{v}_i = \sum_{m=1}^K a_{m,i} v^{(m)}$

This is mathematically equivalent to dynamic, token- and expert-dependent effective projection: $P_{\text{eff},i}(h) = \sum_{m=1}^K a_{m,i}(h) P_m, \quad \hat{v}_i = P_{\text{eff},i}(h) h$

3. Output Aggregation and Computational Invariance

The final routed output combines expert computations with normalized output gates $g_i$: $y = \sum_{i \in \mathcal{S}} g_i E_i\left(\sum_{m=1}^K a_{m,i} v^{(m)}\right)$ where $g_i = \text{Softmax}_{i \in \mathcal{S}}(\bar{s}_i \cdot \alpha_i)$ and $\alpha_i$ is a learned per-expert routing scale.

Crucial Engineering Guarantee:

• Active Experts: Exactly $k$ experts are selected.
• Compute FLOPs: Each selected expert executes exactly one forward pass.
• Communication: Inter-device token dispatch and reduction transfer vectors of size $d_{\text{latent}}$, identical to standard latent-space MoE. The all-to-all communication volume does not increase.

---

Quantitative Evaluation

The authors pretrain a 24-layer 10B-A1B model (10 billion total parameters, 1 billion active parameters per token, top-8 routing with 1 shared expert) on 708 billion tokens.

1. Downstream Benchmark Scores (at 708B Tokens)

Task Category	Benchmark	Baseline (LatentMoE)	Multi-Router ($R=2$)	Multi-View ($K=R=2$)	$\Delta$ (Multi-View vs Baseline)
Exams	CMMLU	53.71%	53.77%	55.20%	+1.49%
	MMLU	57.48%	57.50%	58.67%	+1.19%
	MMLU-Redux	59.03%	58.90%	60.93%	+1.90%
	MMLU-Pro	27.28%	27.71%	27.12%	-0.16%
	Exams Avg.	49.38%	49.47%	50.48%	+1.10%
Knowledge	HotpotQA	22.00%	22.13%	22.54%	+0.54%
	SimpleQA	3.35%	3.85%	3.41%	+0.06%
	PreciseWiki	8.64%	9.16%	9.02%	+0.38%
	Knowledge Avg.	11.33%	11.71%	11.66%	+0.33%
Science	GPQA	30.00%	32.86%	35.60%	+5.60%
Reasoning	BBH	36.10%	37.31%	35.76%	-0.34%
Overall	9-Task Mean	33.07%	33.69%	34.25%	+1.18%

2. Router-Utility Alignment and Training Convergence

• Correlation Improvements: Multi-Router improves the global Spearman rank correlation between router scores and empirical expert utility by +8.1%, and achieves an astounding +58% relative improvement in the deep layer with the worst baseline single-router alignment.
• Late-Stage Loss: Over the final 1,000 steps of pretraining:
  • Baseline loss: $2.25621$
  • Multi-Router loss: $2.25047$ ($\Delta = -5.74 \times 10^{-3}$)
  • Multi-View loss: $2.24883$ ($\Delta = -7.38 \times 10^{-3}$)
• Input Permutation Ablation: Permuting the view-mixing weights $a_{m,i}$ across selected experts increases held-out negative log-likelihood (NLL) by $0.01340$ nats/token, proving that experts actively specialize on token-specific modulated representations.

---

Key Takeaways

1. Routing Should Shape Computation, Not Just Selection: In traditional MoE, routing scores are discarded after selecting experts and scaling their outputs. Multi-View MoE demonstrates that routing preferences can modulate expert input representations directly, unlocking substantial representational diversity.
2. Zero Inefficiency MoE Scaling: Unlike approaches that duplicate computation or require extra matrix multiplications inside FFNs, Multi-View MoE performs convex mixing on pre-dispatch latent vectors. It preserves active parameters, FLOPs, and all-to-all communication budgets.
3. Significant Reasoning Gains on GPQA: The massive +5.60 percentage point jump on GPQA demonstrates that deep multi-step scientific reasoning benefits disproportionately from multi-router disambiguation and expert input specialization.

                    传统 MOE 架构的根本性瓶颈

  隐藏层状态 h
        │
   [单线性路由器] ──> 在中深层，与专家真实输出效用相关性 rho < 0.05！
        │
   Top-k 门控选择
        │
        ├──> 专家 E_1(h)  ──┐
        ├──> 专家 E_2(h)  ──┼──> 加权求和 ──> 输出 y
        └──> 专家 E_k(h)  ──┘
        
  * 痛点：所有被选中的专家接收到的都是完全相同的一张输入特征向量 h，
    路由器在做出选择后，其蕴含的丰富偏好信息即被直接丢弃！

                              MULTI-VIEW MOE 整体架构

                     隐藏层表征 h (维度 d)
                    ┌──────────────┴──────────────┐
                    │                             │
                    ▼                             ▼
        [共享潜在投影基底]                [多路由器子模块]
       P_1 h, P_2 h, ..., P_K h           R 个独立参数化的路由头
          (共 K 个潜在视角)              s_{m,i} = sigmoid(w_m,i^T h)
                    │                             │
                    │               ┌─────────────┴─────────────┐
                    │               ▼                           ▼
                    │         跨路由器求和聚合：         跨路由器归一化
                    │        s_bar_i = sum_m s_m,i       (针对每个专家独立计算):
                    │               │                   a_m,i = s_m,i / sum_n s_n,i
                    │               ▼                           │
                    │       Top-k 选拔集合 S                     │
                    │               │                           │
                    └───────────────┼───────────────────────────┘
                                    │
                                    ▼
                         [潜在视角动态混合]
             针对每个入选专家 i in S 混合输入：
             v_hat_i = sum_{m=1}^K a_m,i * (P_m h)
                                    │
                                    ▼
                         [专家单次前向执行]
             每个激活专家仅执行【一次】前向计算: E_i(v_hat_i)
                                    │
                                    ▼
                         [门控输出加权聚合]
                     y = sum_{i in S} g_i * E_i(v_hat_i)