EMO: Pretraining Mixture of Experts for Emergent Modularity

Paper: 2605.06663 Authors: Ryan Wang, Akshita Bhagia, Sewon Min Institutions: UC Berkeley, Allen Institute for AI Categories: cs.CL Resources:

• Model: hf.co/allenai/EMO
• Code: github.com/allenai/EMO
• Blog: allenai.org/blog/emo
• Visualization: emovisualization.netlify.app

Abstract

Large language models are typically deployed as monolithic systems, requiring the full model even when applications need only a narrow subset of capabilities (e.g., code, math, or domain-specific knowledge). While Mixture-of-Experts (MoEs) activate only a subset of experts per input, restricting inference to expert subsets for a given domain leads to severe performance degradation in practice. This paper introduces EMO, an MoE designed for modularity—the independent use and composition of expert subsets—without requiring human-defined priors. By encouraging tokens from similar domains to rely on similar experts through document-level expert pool constraints, EMO enables coherent expert groupings to emerge during pretraining. A 1B-active, 14B-total parameter EMO trained on 1T tokens matches standard MoE performance while enabling selective expert use: retaining only 25% (12.5%) of experts incurs just 1% (3%) absolute performance drop, whereas standard MoEs break under the same constraints.

Key Contributions

• Emergent Modularity: Introduces a training approach where modular expert structure emerges from data without human-defined domain labels, using only document boundaries as grouping signals.

• Document-Level Expert Pooling: Constrains all tokens within a document to select experts from a shared pool, encouraging domain-coherent expert specialization while allowing different documents to use different pools.

• Practical Memory Efficiency: Enables deployment of only 25% of experts with just 1% performance drop, making large sparse MoEs viable in memory-constrained settings—a critical advantage as models grow larger and sparser.

• Semantic-Level Specialization: Expert subsets specialize at semantic levels (domains like math, code, biomedical) rather than the low-level syntactic patterns (prepositions, punctuation) observed in standard MoEs.

• Composable Architecture: Transforms a single model into a composable system where expert subsets can be independently used and combined, enabling flexible deployment and targeted updates.

Methodology

The Modularity Problem

Standard MoEs face a fundamental limitation:

• Tokens within the same input activate different experts
• Most or all experts are used over the course of a task
• Experts specialize in low-level lexical patterns (e.g., prepositions, punctuation)
• This prevents expert subsets from being usable independently
• Memory-constrained deployment becomes impractical as models grow larger

EMO’s Key Insight

Core Intuition: Tokens from similar domains should activate similar subsets of experts.

Implementation: Since tokens within a document tend to share a domain, EMO restricts all tokens in a document to select their active experts from a shared pool.

Architecture Design

For an MoE with 128 total experts and 8 active experts per token:

• Standard MoE: Each token independently selects 8 experts from all 128
• EMO: All tokens in a document select their 8 active experts from a shared pool of 32 experts

Key properties:

• Different documents may use different expert pools
• No predefined task or domain labels required
• Expert subsets emerge in a self-supervised way
• Document boundaries provide the only grouping signal

Training Configuration

• Model Size: 1B active parameters, 14B total parameters
• Training Data: 1 trillion tokens
• Expert Configuration: 128 total experts, 8 active per token
• Pool Size: 32 experts per document (4x the active count)

Results

Performance with Expert Subsets

EMO enables dramatic memory reduction with minimal performance loss:

Expert Retention	EMO Performance Drop	Standard MoE Drop
100% (full model)	0% (baseline)	0% (baseline)
25% (32 experts)	1% absolute	10% absolute
12.5% (16 experts)	3% absolute	15% absolute

Full Model Performance

As a complete model, EMO matches standard MoE performance across benchmarks, demonstrating that modularity constraints do not sacrifice overall capability.

Domain-Specific Deployment

Across MMLU and MMLU-Pro domain subsets (math, physics, biology, social sciences):

• Identifying and deploying only relevant experts preserves performance
• Memory-accuracy tradeoff significantly improved over standard MoEs
• Expert subsets outperform even fixed-budget models trained from scratch

Expert Specialization Analysis

EMO Expert Specialization:

• Semantic-level: domains and topics (math, code, biomedical)
• Higher-level conceptual patterns
• Coherent groupings aligned with human-interpretable domains

Standard MoE Expert Specialization:

• Syntactic-level: prepositions, punctuation, formatting
• Lower-level lexical patterns
• Fragmented across many experts for any given domain

This qualitative difference in specialization underlies EMO’s modularity and enables selective deployment.

Takeaways

• Modularity as First-Class Objective: EMO demonstrates that modularity can be built into large language models during pretraining, not just as a post-hoc optimization but as a core architectural principle.

• Memory-Constrained Deployment: The ability to deploy only 25% of experts with 1% performance loss has immediate practical implications for serving large sparse models in production, especially on edge devices or memory-limited environments.

• Self-Supervised Domain Discovery: EMO’s approach of using document boundaries alone to induce modular structure is elegant and scalable—no manual domain labeling or task-specific engineering required.

• Semantic vs Syntactic Specialization: The shift from syntactic to semantic expert specialization suggests a fundamentally different learning dynamic, with experts capturing higher-level abstractions rather than surface patterns.

• Composable AI Systems: EMO opens a path toward more flexible AI architectures where capabilities can be independently updated, extended, or composed—imagine updating only the “code” experts without retraining the entire model.

• Interpretability and Control: Semantic-level expert specialization makes models more interpretable and potentially easier to debug, regulate, or align with specific requirements.

• Scaling Implications: As models grow larger and sparser (e.g., DeepSeek-V3 with hundreds of experts), EMO’s approach becomes increasingly valuable for managing memory bottlenecks while maintaining performance.

• Beyond Monolithic Models: This work challenges the monolithic LLM paradigm and demonstrates a viable path toward modular, composable language models that can be tailored to specific deployment contexts.