Training-Free Cross-Architecture Merging for Graph Neural Networks

Paper: 2602.19332 Authors: Rishabh Bhattacharya, Vikaskumar Kalsariya, Naresh Manwani Categories: cs.LG

Abstract

This paper introduces H-GRAMA (Heterogeneous Graph Routing and Message Alignment), a training-free framework that enables merging of Graph Neural Networks with different architectures. Unlike existing model merging techniques that require identical architectures, H-GRAMA operates in operator space rather than parameter space, allowing heterogeneous GNN models like GCN and GAT to be combined without retraining. The framework formalizes Universal Message Passing Mixture (UMPM), which expresses diverse GNN layers through a unified functional representation. Experimental results demonstrate that H-GRAMA maintains specialist model accuracy while achieving 1.2x to 1.9x inference speedups compared to ensemble methods.

Key Contributions

• Introduction of H-GRAMA, the first training-free framework for cross-architecture GNN merging
• Formalization of Universal Message Passing Mixture (UMPM) as a shared operator family that unifies heterogeneous GNN architectures
• Demonstration that operator-space merging overcomes the topology-dependent limitations of parameter-space approaches
• Empirical validation showing preserved specialist accuracy with significant inference speedups over traditional ensembles

The Challenge of GNN Model Merging

Traditional model merging techniques operate in parameter space, averaging or interpolating weights between models with identical architectures. This approach fails for GNNs due to their fundamental reliance on message passing mechanisms that are inherently topology-dependent. Different GNN architectures implement message passing in structurally incompatible ways—GCN uses symmetric normalized aggregation, GAT employs attention-weighted mechanisms, and GraphSAGE applies sampling-based neighborhood aggregation.

The core issue is that direct parameter averaging between, say, a GCN and GAT layer produces meaningless hybrid parameters that don’t correspond to any coherent message passing operation. The attention coefficients in GAT have no semantic correspondence to the fixed aggregation weights in GCN, making naive parameter-space merging fundamentally unreliable for cross-architecture scenarios.

Universal Message Passing Mixture (UMPM)

H-GRAMA’s solution is to lift the merging operation from parameter space to operator space through UMPM. The key insight is that despite architectural differences, all GNN layers can be expressed as compositions of fundamental operations: neighborhood aggregation, feature transformation, and attention/weighting mechanisms.

UMPM defines a universal operator family where each GNN layer $l$ is represented as:

$h_v^{(l)} = \sigma\left(\sum_{u \in \mathcal{N}(v)} \alpha_{uv} W^{(l)} h_u^{(l-1)}\right)$

where $\alpha_{uv}$ captures architecture-specific weighting (uniform for GCN, learned attention for GAT), $W^{(l)}$ represents learnable transformations, and $\sigma$ is the activation function. By decomposing each architecture into this common functional language, H-GRAMA can route and align message passing operations across heterogeneous models.

H-GRAMA Architecture and Routing Mechanism

The framework consists of three core components:

Operator Decomposition: Each specialist GNN model is decomposed into its constituent UMPM operators, preserving the learned parameters but expressing them in the universal representation.

Dynamic Routing: H-GRAMA employs a routing mechanism that selects which specialist operator to apply at each layer based on input characteristics. The routing is deterministic and training-free, using graph structural properties (degree distribution, clustering coefficient) to make routing decisions.

Message Alignment: To ensure compatibility between layers from different architectures, H-GRAMA applies alignment transformations that normalize message scales and dimensions, preventing the accumulation of misalignment errors across layers.

The merged model operates by dynamically selecting operators from the specialist pool at inference time, creating an adaptive computation path through the heterogeneous operator space.

Experimental Results and Performance Analysis

H-GRAMA was evaluated on standard graph benchmarks including Cora, Citeseer, and PubMed for node classification tasks. The experiments merged various architecture pairs: GCN-GAT, GCN-GraphSAGE, and GAT-GraphSAGE.

Key findings include:

Accuracy Preservation: In compatible depth settings (when specialist models have the same number of layers), H-GRAMA retained 95-98% of specialist model accuracy, significantly outperforming naive parameter averaging which degraded to near-random performance.

Inference Speedup: Compared to ensemble methods that require running all specialist models, H-GRAMA achieved 1.2x speedup for 2-model merging and up to 1.9x for 3-model scenarios, as only one operator path is executed per layer.

Depth Sensitivity: Performance degraded when merging models with mismatched depths, highlighting that architectural compatibility extends beyond just the operator types to include structural properties like network depth.

Routing Efficiency: The training-free routing mechanism successfully identified appropriate operators for different graph regions, with high-degree nodes benefiting from attention-based operators (GAT) and low-degree nodes from simpler aggregation (GCN).

Takeaways

1. H-GRAMA demonstrates that cross-architecture model merging is feasible for GNNs when approached through operator space rather than parameter space
2. The Universal Message Passing Mixture provides a principled framework for expressing heterogeneous GNN architectures in a common functional language
3. Training-free merging can achieve significant inference speedups (1.2x-1.9x) over ensembles while maintaining high accuracy
4. Architectural compatibility for merging extends beyond operator types to include structural properties like network depth
5. Dynamic routing based on graph structural properties enables adaptive computation that leverages specialist strengths for different input characteristics