Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learning

TL;DR

This paper introduces NSG-MoE, a novel multimodal graph learning framework that decomposes nodes into modality-specific parts and employs relation-aware experts, effectively handling modality confusion and improving performance on benchmarks.

Contribution

The paper proposes a new framework combining node splitting, graph rewiring, and Mixture-of-Experts to better process multimodal graphs and mitigate modality confusion issues.

Findings

NSG-MoE outperforms strong baselines on three benchmarks.

The method achieves competitive training efficiency despite using MoE.

Spectral analysis shows adaptive filtering over modality-specific subspaces.

Abstract

Multimodal graphs are gaining increasing attention due to their rich representational power and wide applicability, yet they introduce substantial challenges arising from severe modality confusion. To address this issue, we propose NSG (Node Splitting Graph)-MoE, a multimodal graph learning framework that integrates a node-splitting and graph-rewiring mechanism with a structured Mixture-of-Experts (MoE) architecture. It explicitly decomposes each node into modality-specific components and assigns relation-aware experts to process heterogeneous message flows, thereby preserving structural information and multimodal semantics while mitigating the undesirable mixing effects commonly observed in general-purpose GNNs. Extensive experiments on three multimodal benchmarks demonstrate that NSG-MoE consistently surpasses strong baselines. Despite incorporating MoE -- which is typically computationally heavy -- our method achieves competitive training efficiency. Beyond empirical results, we provide a spectral analysis revealing that NSG performs adaptive filtering over modality-specific subspaces, thus explaining its disentangling behavior. Furthermore, an information-theoretic analysis shows that the architectural constraints imposed by NSG reduces mutual information between data and parameters and improving generalization capability.