OptiMAG: Structure-Semantic Alignment via Unbalanced Optimal Transport

TL;DR

OptiMAG introduces an unbalanced optimal transport regularization to align semantic and structural information in multimodal attributed graphs, improving performance across various graph and multimodal tasks.

Contribution

It proposes a novel regularization framework using Fused Gromov-Wasserstein distance for structural-semantic alignment in multimodal graphs, adaptable as a plug-in for existing models.

Findings

Outperforms baselines on node classification and link prediction.

Enhances multimodal generation tasks like graph2text and graph2image.

Effectively mitigates structural-semantic conflicts in MAGs.

Abstract

Multimodal Attributed Graphs (MAGs) have been widely adopted for modeling complex systems by integrating multi-modal information, such as text and images, on nodes. However, we identify a discrepancy between the implicit semantic structure induced by different modality embeddings and the explicit graph structure. For instance, neighbors in the explicit graph structure may be close in one modality but distant in another. Since existing methods typically perform message passing over the fixed explicit graph structure, they inadvertently aggregate dissimilar features, introducing modality-specific noise and impeding effective node representation learning. To address this, we propose OptiMAG, an Unbalanced Optimal Transport-based regularization framework. OptiMAG employs the Fused Gromov-Wasserstein distance to explicitly guide cross-modal structural consistency within local neighborhoods, effectively mitigating structural-semantic conflicts. Moreover, a KL divergence penalty enables adaptive handling of cross-modal inconsistencies. This framework can be seamlessly integrated into existing multimodal graph models, acting as an effective drop-in regularizer. Experiments demonstrate that OptiMAG consistently outperforms baselines across multiple tasks, ranging from graph-centric tasks (e.g., node classification, link prediction) to multimodal-centric generation tasks (e.g., graph2text, graph2image). The source code will be available upon acceptance.