MobileMoE: Bringing Mixture-of-Experts to Your Smartphone

Paper: 2605.27358 Authors: Yanbei Chen, Hanxian Huang, Ernie Chang, Jacob Szwejbka, Digant Desai, Zechun Liu, Vikas Chandra, Raghuraman Krishnamoorthi Categories: cs.LG, cs.AI, cs.CL

The Gap

Mixture-of-Experts (MoE) works brilliantly at 100B+ parameters—GPT-4, Mixtral, and others use it to scale efficiently. But nobody knew if MoE made sense below 1B parameters, especially on phones where memory is tight and every millisecond counts. The conventional wisdom: MoE’s routing overhead and memory fragmentation would kill performance at small scales. Dense models like MobileLLM dominated the sub-billion regime.

The gap: Does MoE’s conditional computation advantage survive when you shrink the model to fit in 2-4GB of phone memory and optimize for ARM chips?

Problem: On-device LLMs need speed + quality in <1B active params
   |
   v
Assumption: MoE's sparse activation can beat dense models
            even under mobile memory/compute constraints
   |
   v
Method: Derive mobile-specific MoE scaling laws
        -> Find "sweet spot" architecture (moderate sparsity, fine-grained experts)
        -> Train with 4-stage recipe + quantization-aware training
   |
   v
Evidence: 2-4x faster inference than dense baselines
          Matches quality with 60% fewer total parameters
   |
   v
Conclusion: MoE is viable and superior for sub-billion on-device LLMs

The Increment

One sentence: Before this paper, MoE was a datacenter-scale technique; after, it’s a proven architecture for phone-based LLMs with concrete deployment recipes.

Core Mechanism

MobileMoE starts by formulating scaling laws specifically for mobile constraints. Unlike datacenter MoE that optimizes for training efficiency, this paper jointly optimizes for memory footprint (model must fit in phone RAM) and inference compute (latency on ARM CPUs). The key finding: moderate sparsity (activating 25-33% of experts) with fine-grained experts (many small experts instead of few large ones) hits the sweet spot. Too sparse and routing overhead dominates; too dense and you lose the speed advantage.

The architecture uses shared experts—a subset of experts that always activate—alongside routed experts. This design reduces memory fragmentation (shared experts stay in cache) and stabilizes training. Each token routes to top-k experts per layer, but unlike datacenter MoE that might use 8-of-128 routing, MobileMoE uses 2-of-8 or 3-of-12 patterns optimized for mobile memory bandwidth.

Training follows a four-stage recipe: (1) pre-training on web text, (2) mid-training on curated data to improve instruction-following, (3) supervised fine-tuning on instruction datasets, (4) quantization-aware training to INT4 weights. The last stage is critical—naive quantization destroys MoE routing decisions, so they retrain the router and experts jointly under quantization noise.

Input Token
    |
    v
  Router (learns which experts to activate)
    |
    +---> Shared Experts (always active, stay in cache)
    |
    +---> Top-K Routed Experts (conditionally active)
          |
          +---> Expert 1 (FFN)
          +---> Expert 2 (FFN)
          +---> Expert 3 (FFN)
          ...
    |
    v
  Weighted Sum of Expert Outputs
    |
    v
  Next Layer

Think of MobileMoE like a restaurant kitchen with specialized stations. A dense model is one chef doing everything—slow but simple. A datacenter MoE is a massive kitchen with 128 stations where each dish visits 8 stations—fast but needs huge space. MobileMoE is a food truck: 8-12 compact stations, each dish visits 2-3, plus a prep station (shared experts) that handles every order. The dispatcher (router) is trained to send orders to the right stations without causing congestion. The food truck constraint forces smart specialization—you can’t have 128 stations, so each station must be genuinely useful, and the dispatcher must be accurate because wrong routing wastes precious time.

Key Concepts

• Moderate Sparsity: In MoE, sparsity means what fraction of the model activates per token. Datacenter MoE uses extreme sparsity (activate 5-10% of parameters) to scale to trillions of parameters. But on phones, extreme sparsity backfires—routing overhead (deciding which experts to use) becomes a larger fraction of total compute, and memory access patterns get chaotic. Moderate sparsity (25-33% activation) keeps routing cheap and memory access predictable. Imagine a library: extreme sparsity is retrieving 1 book from 1000 shelves (lots of walking), moderate sparsity is retrieving 3 books from 12 shelves (less overhead, still specialized).

• Fine-Grained Experts: Instead of 8 large experts (each expert is a big neural network), use 12 smaller experts. Why? Memory bandwidth. On ARM chips, loading one giant expert from DRAM is slower than loading two small experts because you can pipeline the loads. Fine-grained experts also improve specialization—12 experts can cover more distinct patterns than 8. The tradeoff: more routing decisions, but the paper shows this is negligible compared to memory savings.

• Quantization-Aware Training for MoE: Standard quantization (converting FP16 weights to INT4) assumes all weights matter equally. But in MoE, the router’s decisions are fragile—small weight changes can flip which expert activates, cascading into quality loss. Quantization-aware training simulates INT4 noise during training, so the router learns to make robust decisions even with coarse weights. It’s like training a pilot in a flight simulator with turbulence instead of only smooth conditions—the pilot (router) learns to handle the noise (quantization error) that will exist in deployment.

Framework Shift

Before (Dense LLMs):                After (MobileMoE):
                                    
Input --> Layer 1 --> Layer 2       Input --> Router --> Shared + Top-K Experts
          (all params active)                  |              |
          |                                    v              v
          v                               (25-33% active) (always active)
       Output                                  |
                                               v
       Memory: 100%                         Output
       Compute: 100%                        
                                            Memory: 100% (total params)
                                            Compute: 25-33% (active params)

From “activate everything every time” to “activate what you need, keep essentials always ready”—the core shift is conditional computation with mobile-aware routing.

Expert Assessment

Problem choice: Real gap. On-device LLMs are exploding (Apple Intelligence, Gemini Nano, etc.) but everyone’s using dense models because MoE seemed too complex for mobile. This paper directly challenges that assumption with a well-motivated question: can we get MoE’s efficiency at sub-billion scale?

Method maturity: Solid engineering with some clever insights. The mobile-specific scaling laws are genuinely useful—they don’t just port datacenter MoE, they rethink the architecture under different constraints. The four-stage training recipe is thorough but not groundbreaking (it’s standard practice applied carefully). The quantization-aware training for MoE is a nice touch that shows attention to deployment reality. No major shortcuts, though the paper could have explored dynamic expert selection (adapting sparsity per input) more deeply.

Experimental integrity: Strong baselines and fair comparisons. They compare against MobileLLM (dense baseline from the same team) and OLMoE (state-of-the-art open MoE), both reasonable choices. The 14 benchmarks cover reasoning, knowledge, and instruction-following. On-device profiling on actual Pixel phones is rare and valuable—most papers stop at FLOPs. One quibble: they don’t report training cost or carbon footprint, which matters for reproducibility. The numbers hold up, no obvious cherry-picking.

Writing quality: Clear and well-structured, but the scaling law section (Section 3) is dense and could use more intuition before diving into equations. The related work is thorough but reads like a checklist. The deployment section (Section 6) is excellent—concrete latency numbers, memory breakdowns, and profiling insights. If they rewrote Section 3 to lead with intuition (why moderate sparsity? show the tradeoff curve first, then formalize), the paper would be much more accessible.

Verdict: Strong accept—this is solid systems research that solves a real problem with careful engineering and thorough evaluation. It’s not a conceptual breakthrough, but it’s the kind of work that moves the field forward by making a technique practical in a new regime.

Takeaways

For practitioners building on-device models: The moderate sparsity insight transfers beyond MoE. If you’re designing any conditional computation system for mobile (early exit, dynamic depth, adaptive width), aim for 25-35% sparsity—it’s the sweet spot where routing overhead is manageable and memory access stays predictable.

For MoE researchers: Quantization-aware training for routers is underexplored. Most MoE papers quantize post-hoc and report quality loss. This paper shows joint training of routers and experts under quantization noise is essential for deployment. Steal this for any MoE system targeting INT8 or INT4.

For mobile ML engineers: The four-stage training recipe (pre-train → mid-train → SFT → QAT) is a practical template. Mid-training on curated data before instruction tuning is often skipped but clearly helps here. The on-device profiling methodology (Section 6) is a good reference for how to report mobile performance beyond FLOPs.

Transferable technique: Fine-grained experts with shared experts. This architectural pattern (many small specialists + always-on generalists) is applicable to any resource-constrained conditional computation system, not just LLMs. Think recommendation systems, on-device vision models, or even compiler optimization passes.