FaaSMoE: Serverless Mixture-of-Experts for Multi-Tenant AI Serving

Paper: 2604.26881 Authors: Minghe Wang, Trever Schirmer, Mohammadreza Malekabbasi, David Bermbach Categories: cs.DC, cs.LG

The Gap

Mixture-of-Experts (MoE) models activate only a small subset of experts per input—say 2 out of 64—but traditional deployments keep all experts loaded in memory. This creates massive resource waste: you provision for 64 experts but use 2 at a time. In multi-tenant scenarios where different users trigger different experts, the waste compounds. Prior work (DeepSpeed-MoE, Tutel, MegaBlocks) optimizes expert parallelism and routing but assumes all experts stay resident. No one has seriously questioned the “all experts in memory” assumption for serving workloads.

Problem: MoE models waste memory
         (provision 64 experts, use 2)
            |
            v
Assumption: FaaS scale-to-zero can match
            MoE's sparse activation pattern
            |
            v
Method: Deploy each expert as stateless
        FaaS function + router as gateway
            |
            v
Evidence: 67% resource reduction on
          Qwen1.5-MoE-2.7B workload
            |
            v
Conclusion: Serverless MoE viable for
            multi-tenant edge deployment

The Increment

One sentence: Before this paper, MoE serving meant keeping all experts in memory; after, you can deploy experts as on-demand functions that scale to zero between requests.

Core Mechanism

FaaSMoE splits MoE into two planes. The control plane runs a lightweight router that receives input tokens, computes expert routing scores, and dispatches requests to the appropriate expert functions. The execution plane consists of expert models deployed as stateless FaaS functions—each function loads its expert weights on cold start, processes the routed tokens, and returns hidden states back to the router. The router aggregates expert outputs and passes them to the next layer.

The key design choice is expert granularity. You can deploy one expert per function (fine-grained) for maximum elasticity, or pack multiple experts into one function (coarse-grained) to amortize cold start overhead. FaaSMoE makes this configurable: a “granularity factor” k means each function hosts k experts. When k=1, you get per-expert scaling; when k equals total experts, you’re back to monolithic deployment.

Think of it like a restaurant kitchen during off-peak hours. Traditional MoE is keeping all stations staffed even when orders are sparse—the grill, the fryer, the salad station all have cooks standing by. FaaSMoE is calling in cooks only when orders arrive for their station. The expeditor (router) reads each ticket, calls the relevant cook (expert function), and assembles the dish (aggregates outputs). If you’re worried about call-in time, you can assign each cook multiple stations (higher granularity k) so fewer calls are needed, trading some idle time for faster response.

Key Concepts

• Expert Activation Sparsity: In MoE models, each input token is routed to a small subset of experts—typically 1-2 out of 8-64 total experts. The router computes a score for each expert and picks the top-k. This means 95%+ of experts sit idle for any given token, yet traditional deployments keep them all loaded. FaaSMoE exploits this sparsity: if an expert isn’t being called, it doesn’t consume resources.

• Cold Start Amortization: FaaS functions have cold start latency—the first invocation loads the model into memory. For small experts (~100MB), this takes 1-2 seconds. FaaSMoE mitigates this by (1) keeping frequently-used experts warm through request patterns, and (2) allowing multiple experts per function so one cold start loads several experts at once. The paper shows that with granularity k=4, cold starts drop to ~15% of requests under realistic workloads.

• Multi-Tenant Resource Sharing: When multiple users share an MoE deployment, their requests trigger different expert subsets. User A’s queries might favor experts 1, 5, 12; User B’s favor 3, 8, 15. Traditional deployments provision for the union of all experts across all users. FaaSMoE provisions only for the active working set at any moment—if User A and B’s requests don’t overlap in time, their expert functions don’t overlap in memory.

Framework Shift

Before (monolithic MoE):              After (FaaSMoE):

[Input] --> [Router]                  [Input] --> [Router/Gateway]
              |                                      |
              v                                      v
    +-------------------+               +-------------------------+
    | All 64 Experts    |               | FaaS Platform           |
    | Loaded in Memory  |               |   Expert-1 (cold)       |
    |                   |               |   Expert-5 (warm) <--+  |
    | Expert-1  Expert-2|               |   Expert-12 (cold)    |  |
    | Expert-3  Expert-4|               |   ...                 |  |
    | ...       ...     |               +-------------------------+
    | Expert-63 Expert-64               Only invoked experts
    +-------------------+               consume resources
    Always consuming                    
    full memory footprint               

From static provisioning to demand-driven allocation, the core shift is treating experts as ephemeral compute rather than persistent state.

Expert Assessment

Problem choice: Real gap. Multi-tenant MoE serving is an emerging pain point as these models move from research to production. The memory waste is measurable and the serverless angle is underexplored. However, the problem is somewhat narrow—it matters most for edge deployments and smaller MoE models where cold start overhead is tolerable.

Method maturity: Straightforward engineering rather than algorithmic novelty. The insight is architectural: recognizing that MoE’s sparse activation pattern maps naturally to FaaS’s scale-to-zero model. The granularity knob is practical but not surprising. The paper doesn’t address harder questions like expert placement optimization or predictive warming strategies.

Experimental integrity: Baselines are fair but limited. They compare against full-model deployment and show 67% resource reduction, which is believable given the activation sparsity. However, they only test one model (Qwen1.5-MoE-2.7B) and one FaaS platform (OpenFaaS). No comparison against other MoE serving systems like DeepSpeed or Tutel in multi-tenant mode. The workload is synthetic—real production traffic patterns might behave differently. Cold start numbers (1-2s) are measured but not deeply analyzed.

Writing quality: The paper is clear but shallow. Section 3 (design) reads like a system description rather than a design rationale—why these choices over alternatives? Section 4 (evaluation) reports numbers without much interpretation. The related work section misses recent MoE serving papers from late 2023. If they rewrote Section 5 (discussion) to include failure modes, cost analysis, and comparison with expert caching strategies, the paper would be much stronger.

Verdict: weak accept — Solid systems work addressing a real problem, but limited scope and shallow evaluation prevent it from being a strong contribution.

Takeaways

The core idea transfers beyond MoE: any model with sparse activation (conditional computation, early exit, dynamic depth) can benefit from function-based deployment. The granularity knob is worth stealing—it’s a simple way to trade elasticity for overhead in any disaggregated system.

For practitioners: if you’re serving MoE models with bursty multi-tenant traffic and can tolerate 1-2s cold starts, this architecture cuts your memory bill significantly. If you need sub-100ms latency or serve a single high-throughput tenant, stick with traditional deployment.

The paper also hints at a broader pattern: as models grow more modular (MoE, retrieval-augmented, tool-using), serverless primitives become more applicable. The challenge is managing the control plane overhead—routing, aggregation, state management—which this paper doesn’t fully solve.