Training-Free Cross-Architecture Merging for Graph Neural Networks

TL;DR

H-GRAMA introduces a training-free method for merging heterogeneous GNN architectures by aligning message passing operators, enabling cross-architecture integration without retraining and improving inference efficiency.

Contribution

The paper proposes H-GRAMA, a novel operator-space merging framework for heterogeneous GNNs that avoids retraining and maintains high accuracy.

Findings

Enables cross-architecture GNN merging (e.g., GCN to GAT) without retraining.

Achieves inference speedups of 1.2x to 1.9x over ensembles.

Retains high specialist accuracy in compatible depth settings.

Abstract

Model merging has emerged as a powerful paradigm for combining the capabilities of distinct expert models without the high computational cost of retraining, yet current methods are fundamentally constrained to homogeneous architectures. For GNNs, however, message passing is topology-dependent and sensitive to misalignment, making direct parameter-space merging unreliable. To bridge this gap, we introduce H-GRAMA (Heterogeneous Graph Routing and Message Alignment), a training-free framework that lifts merging from parameter space to operator space. We formalize Universal Message Passing Mixture (UMPM), a shared operator family that expresses heterogeneous GNN layers in a common functional language. H-GRAMA enables cross-architecture GNN merging (e.g., GCN to GAT) without retraining, retaining high specialist accuracy in most cases in compatible depth settings and achieving inference speedups of 1.2x to 1.9x over ensembles.