Model Merging in LLMs: A Comprehensive Survey Through the FUSE Framework

Paper: 2603.09938 Authors: Mingyang Song, Mao Zheng Categories: cs.CL

The Gap

Model merging techniques have exploded alongside the proliferation of fine-tuned LLMs. You’ve got weight averaging, task arithmetic, mixture-of-experts, evolutionary methods—all scattered across different papers with different motivations. A practitioner looking to combine a safety-aligned model with a domain-specialized one faces a bewildering menu of options with no clear map of when to use what. Prior surveys either focused narrowly on specific techniques or predated the LLM era entirely. The field needed a unifying framework to organize this landscape.

Problem: Scattered merging techniques, unclear relationships
    |
    v
Assumption: Techniques can be organized along systematic dimensions
    |
    v
Method: FUSE taxonomy (Foundations/Unification/Scenarios/Ecosystem)
    |
    v
Evidence: Comprehensive coverage of 100+ papers organized coherently
    |
    v
Conclusion: Structured understanding enables systematic method selection

The Increment

One sentence: Before this paper, model merging was a grab bag of techniques; after, practitioners have a four-dimensional map showing where each method sits and when to reach for it.

Core Mechanism

The FUSE framework isn’t a new merging algorithm—it’s an organizational schema. The F (Foundations) layer establishes theoretical grounding: why can we even merge models? This covers loss landscape geometry, mode connectivity, and the linear mode connectivity hypothesis that suggests fine-tuned models from the same base live in a connected region of weight space. The U (Unification Strategies) layer catalogs the actual algorithms: simple weight averaging, task vector arithmetic (treating fine-tuning as a direction in weight space), sparsification methods that merge only important parameters, MoE architectures that route inputs to specialized experts, and evolutionary approaches that search for optimal merge configurations.

The S (Scenarios) layer maps methods to applications: multi-task learning, safety alignment, domain adaptation, multilingual transfer, federated learning. Each scenario has different constraints—safety alignment can’t compromise on harmful outputs, multilingual transfer needs to preserve language-specific knowledge. The E (Ecosystem) layer surveys the infrastructure: mergekit, PEFT libraries, model hubs, evaluation benchmarks. This layer acknowledges that merging isn’t just algorithms—it’s a sociotechnical practice enabled by tools and communities.

FUSE Taxonomy Structure:

    [Foundations]  <-- Theory: why merging works
         |
         v
  [Unification]   <-- Methods: weight avg, task vectors, MoE, etc.
         |
         v
    [Scenarios]    <-- Applications: multi-task, safety, domain, etc.
         |
         v
    [Ecosystem]    <-- Tools: mergekit, benchmarks, platforms

Think of FUSE like organizing a city’s transportation system. Foundations is understanding the terrain—where are the hills, rivers, population centers? You can’t plan routes without knowing the geography. Unification Strategies are the actual transportation modes—buses, trains, bikes, cars. Each has different speed/capacity/cost tradeoffs. Scenarios are the use cases—commuting to work needs different solutions than moving freight or emergency response. Ecosystem is the infrastructure—roads, stations, maintenance facilities, and the community of drivers and riders who make it all work. Just as you wouldn’t choose a transportation mode without considering terrain, destination, and available infrastructure, you shouldn’t pick a merging method without considering the theoretical constraints, application requirements, and tooling support.

Key Concepts

• Model Merging: Instead of training a new model from scratch or running multiple models in parallel (ensemble), you combine the weights of pre-trained models into a single model. The key insight: if models share a common base (like the same pre-trained LLM), their weight spaces are related. Merging is like blending paints—you’re not creating new colors from scratch, you’re mixing existing ones. The challenge: unlike paints, neural network weights interact nonlinearly, so naive averaging often fails. Advanced methods handle this through sparsification (only merge important weights), task vectors (merge along meaningful directions), or routing (let different parts of the model specialize).

• Linear Mode Connectivity: Imagine two fine-tuned models as points in a high-dimensional weight space. Linear mode connectivity says you can draw a straight line between them, and every point on that line is also a good model (low loss). This is surprising—neural networks are highly nonlinear, so you’d expect the path between two solutions to pass through terrible models. But empirically, models fine-tuned from the same base often exhibit this property. Why does this matter? It means simple interpolation (weighted averaging) can work. If the path between models were jagged, you’d need sophisticated search methods.

• Task Vectors: When you fine-tune a base model for a specific task, the weight changes form a vector in weight space. This vector captures “what the model learned” for that task. Task arithmetic treats these vectors like regular vectors—you can add them (combine capabilities), subtract them (remove unwanted behaviors), or scale them (control strength). Example: if you have a base model, a math-tuned version, and a code-tuned version, you can compute task_vector_math = weights_math - weights_base and task_vector_code = weights_code - weights_base, then create a math+code model via weights_base + task_vector_math + task_vector_code. The magic: this often works without any additional training.

Framework Shift

Before (scattered landscape):        After (FUSE taxonomy):

Papers:                              Organized view:
[Method A]  [Method B]               
[Method C]  [Method D]               [Foundations] ----+
[Method E]  [Method F]                    |           |
     ...                                  v           v
                                   [Unification] [Scenarios]
Practitioner:                            |           |
"Which one do I use?"                    v           v
"How do they relate?"                [Ecosystem]  [Applications]
"What are the tradeoffs?"            
                                     Practitioner:
                                     "I need multi-task + safety"
                                     "FUSE says: task vectors + 
                                      sparsification, check mergekit"

From scattered techniques to structured navigation, the core shift is from ad-hoc selection to systematic method matching based on theoretical constraints, application requirements, and available tooling.

Expert Assessment

Problem choice: This is a real gap. Model merging has gone from niche trick to practical necessity as organizations accumulate dozens of fine-tuned models. The timing is right—the field has matured enough to benefit from organization but hasn’t ossified into established categories. A survey now can shape how people think about the space.

Method maturity: The FUSE taxonomy is clean but not revolutionary. Four dimensions (theory/methods/applications/tools) is a natural decomposition. The value is in execution—did they actually cover the landscape comprehensively? Did they identify meaningful subcategories within each dimension? The abstract mentions 100+ papers, which suggests decent coverage. The real test: does a practitioner finish reading and know which method to try first?

Experimental integrity: N/A for a survey, but the quality hinges on whether the taxonomy is actually useful or just another acronym. Red flag to watch: do the authors force-fit papers into FUSE categories, or does the framework emerge naturally from the literature?

Writing quality: Survey papers live or die by their organization. If the FUSE framework is front-loaded clearly with a visual roadmap, readers can navigate. If it’s buried in dense prose, the survey becomes a literature dump. The abstract is well-structured, which is promising. The weakest section is likely the Ecosystem—tools and benchmarks date quickly and feel like appendices. If I were revising, I’d make the Scenarios section more prescriptive: “If you have X constraints and Y goals, start with Z method family.”

Verdict: Weak accept — Useful organizational contribution for a rapidly growing area, but the impact depends on whether FUSE becomes a shared vocabulary or just another framework that gets cited once and forgotten.

Takeaways

For practitioners: The FUSE framework gives you a decision tree. Start with your scenario (multi-task? safety? domain?), check what theoretical constraints apply (do your models share a base? are they in the same modality?), then filter methods accordingly. The Ecosystem section should be your first stop—if mergekit already implements what you need, don’t reinvent it.

For researchers: The gaps identified in the survey are your research agenda. The authors likely call out areas where theory lags practice (why does task arithmetic work so well empirically?) and where methods are immature (scaling to models with different architectures). These are your opportunities.

Concrete steal: The task vector formulation is immediately applicable. If you have a base model and multiple fine-tuned versions, you can experiment with task arithmetic in an afternoon. Compute the task vectors, try different linear combinations, and see what emerges. No additional training required. This is the lowest-hanging fruit in the entire survey.