Shodh-MoE: Solving Multi-Physics Interference with Sparse Expert Routing

Paper: 2605.15179 Authors: Ellwil Sharma, Arastu Sharma Categories: cs.LG, cs.AI, physics.comp-ph

The Gap

Universal foundation models for scientific computing promise to handle multiple physical regimes in one network. But when you train a single dense neural operator on both open-channel fluid dynamics and porous media flows, the gradients fight each other. Open-channel flows need spectral representations for turbulent cascades. Porous media flows need geometric sensitivity for boundary-dominated transport. Forcing both through the same parameter path causes gradient conflict, unstable training, and plasticity loss—the model forgets what it learned about one regime when learning the other.

Prior work either trains separate models per physics domain (no knowledge transfer) or uses dense multi-task architectures that suffer from negative transfer. The gap: no architecture that lets different physics regimes coexist in one model without interfering.

Problem: Multi-physics co-training
    |
    v
Gradient conflict in dense networks
    |
    v
Assumption: Different physics need different parameter paths
    |
    v
Method: Sparse MoE routing to specialized experts
    |
    v
Evidence: Autonomous domain bifurcation in routing telemetry
    |
    v
Conclusion: Sparse routing mitigates multi-physics interference

The Increment

One sentence: Before this paper, multi-physics neural operators either trained separately or suffered gradient conflict; after, a single model routes different regimes to specialized experts and converges simultaneously.

Core Mechanism

Shodh-MoE compresses 3D physical fields into 16³ latent tokens using a physics-informed autoencoder. The encoder enforces divergence-free velocity through a Helmholtz decomposition inside the tokenizer—velocity is parameterized as the curl of a vector potential, guaranteeing zero divergence by construction. This produces physically valid latent representations.

A transformer processes these latent tokens, but instead of routing all tokens through the same feedforward layers, a Top-1 soft-semantic router examines each token and assigns it to one of several expert subnetworks. Each expert is a separate feedforward block with its own parameters. The router learns to send tokens from different physical regimes to different experts. Some experts can be shared across regimes for universal symmetries, but domain-specific experts handle regime-specific mechanisms.

During training on mixed open-channel and porous-media data, the router autonomously learns to bifurcate: open-channel tokens go to Expert 0, porous-media tokens go to Expert 1. No manual labeling of token types—the router discovers this split from gradient signals alone. The model converges on both regimes simultaneously because each expert’s gradients no longer conflict with the other’s.

Input: 3D physical field (128^3 grid)
    |
    v
Physics-informed autoencoder
    | (Helmholtz velocity parameterization)
    v
Compressed latent tokens (16^3)
    |
    v
    +---> Router (examines each token)
    |         |
    |         +---> Expert 0 (open-channel specialist)
    |         |
    |         +---> Expert 1 (porous-media specialist)
    |         |
    |         +---> Shared experts (universal symmetries)
    |
    v
Transformer output (processed latents)
    |
    v
Decoder (reconstructs physical field)
    |
    v
Output: Divergence-free velocity field

Think of this like a hospital emergency room. Patients (tokens) arrive with different conditions (physical regimes). A triage nurse (router) quickly assesses each patient and sends them to the appropriate specialist: cardiac patients to the cardiologist (Expert 0), trauma cases to the trauma surgeon (Expert 1). Some staff like nurses (shared experts) help everyone. Each specialist develops deep expertise in their domain without being distracted by unrelated cases. The triage nurse learns which symptoms map to which specialist purely from observing outcomes—no one hands her a manual. The hospital (model) handles all cases simultaneously without specialists stepping on each other’s toes.

Key Concepts

• Negative transfer: When training a neural network on multiple tasks simultaneously makes it worse at each task than training separate networks. Imagine learning piano and drums at the same time—if practicing piano makes you worse at drums, that’s negative transfer. In multi-physics models, gradients from one PDE regime can push parameters in directions that hurt performance on another regime. The network oscillates between regimes instead of improving on both.

• Divergence-free velocity: In incompressible fluid flow, mass conservation requires that velocity divergence equals zero (∇·v = 0). If you imagine velocity as arrows showing flow direction, divergence measures whether flow is “spreading out” (positive) or “converging” (negative) at each point. Zero divergence means flow is neither created nor destroyed—what flows in must flow out. Standard neural networks can output any velocity field, including physically impossible ones with non-zero divergence. Shodh-MoE enforces this constraint by parameterizing velocity as v = ∇ × A (curl of a vector potential), which mathematically guarantees ∇·v = 0 because the divergence of a curl is always zero.

• Sparse activation: In a mixture-of-experts model, “sparse” means each input activates only a subset of experts, not all of them. If you have 8 experts but each token only goes to 1 expert (Top-1 routing), then 7/8 of the expert parameters are inactive for that token. This is opposite to dense models where every parameter processes every input. Sparse activation enables specialization—experts can develop distinct skills because they see different subsets of data—and reduces computation because you’re not running the full model for every input.

Framework Shift

Before (dense multi-task):           After (sparse MoE):

All physics                          Open-channel tokens
    |                                    |
    v                                    v
+-------------------+                +--------+
| Single parameter  |                | Expert |
| path (all layers  |                |   0    |
| process all data) |                +--------+
+-------------------+                    
    |                                Porous-media tokens
    v                                    |
Gradient conflict                        v
Unstable training                    +--------+
                                     | Expert |
                                     |   1    |
                                     +--------+
                                         
                                     No gradient conflict
                                     Simultaneous convergence

From forcing all physics through one parameter bottleneck to giving each regime its own specialized path, the core shift is from shared interference to isolated specialization with selective sharing.

Expert Assessment

Problem choice: Real gap. Multi-physics foundation models are a legitimate frontier in scientific ML, and negative transfer is a documented blocker. The problem sits at the intersection of scaling laws (bigger models, more tasks) and physical constraints (conservation laws, regime-specific mechanisms). Not manufactured—this is where the field is stuck.

Method maturity: Clever architectural insight, not brute force. The Helmholtz velocity parameterization is elegant—enforcing divergence-free constraints inside the tokenizer rather than as a post-hoc loss term. The sparse routing is borrowed from NLP (Switch Transformers, GLaM) but the application to multi-physics is novel. However, the paper doesn’t explore simpler baselines like task-specific adapter layers or low-rank expert decompositions. The jump to full MoE might be overkill for two regimes.

Experimental integrity: Strong on physics verification (divergence ~10⁻¹⁰ is impressive), but the experimental setup is narrow. Only two physical regimes, only 20,000 training steps, no comparison to other multi-task architectures beyond implicit claims about dense models. The routing telemetry (Expert 0 for open-channel, Expert 1 for porous-media) is compelling evidence of autonomous bifurcation, but we don’t see what happens with more than two regimes or with regimes that share more overlap. Baselines are missing—no comparison to separate models, no ablation on number of experts, no analysis of what happens if you force dense routing.

Writing quality: The abstract and introduction are crisp. The method section buries the key insight (Helmholtz parameterization) in a subordinate clause—this should be front and center. The results section reports numbers without error bars or statistical significance tests. The related work section is thin—doesn’t engage with recent work on modular networks, task arithmetic, or other approaches to negative transfer. Rewriting the method section to lead with the physics constraints and then show how the architecture satisfies them would elevate the whole paper.

Verdict: weak accept — Solid architectural contribution with strong physics verification, but experimental scope is too narrow and baselines are missing. The core idea (sparse routing for multi-physics) is sound and the divergence-free guarantee is valuable, but the paper needs broader evaluation to support claims about “universal” foundation models.

Takeaways

Enforce physical constraints in the architecture, not the loss function. The Helmholtz velocity parameterization guarantees divergence-free flow by construction—no need to add a penalty term and hope the optimizer respects it. This pattern transfers: if your domain has hard constraints (symmetries, conservation laws, boundary conditions), bake them into the network structure rather than treating them as soft objectives.

Routing telemetry as a diagnostic tool. Watching which tokens go to which experts during training reveals whether the model is learning meaningful specialization or just random assignment. If held-out validation tokens from different regimes route to different experts without supervision, that’s evidence the architecture is doing something right. This applies beyond physics—any multi-domain model can use routing patterns to diagnose whether domains are interfering.

Sparse activation for gradient isolation. When tasks have conflicting gradients, sparse routing lets each task update its own parameters without stepping on the others. This is cheaper than training separate models and more effective than dense multi-task learning. The tradeoff: routing overhead and potential load imbalance (some experts get more traffic than others). Worth exploring in any setting where negative transfer is a problem.