Unveiling Mode Connectivity in Graph Neural Networks

TL;DR

This paper explores the geometric properties of loss landscapes in GNNs through mode connectivity, revealing how graph structure influences optimization and generalization, and proposing new diagnostic tools and training insights.

Contribution

It is the first to investigate mode connectivity in GNNs, highlighting the role of graph properties over architecture in loss landscape behavior.

Findings

GNNs show unique non-linear mode connectivity patterns.

Graph properties like homophily influence mode connectivity.

Mode connectivity relates to GNN generalization performance.

Abstract

A fundamental challenge in understanding graph neural networks (GNNs) lies in characterizing their optimization dynamics and loss landscape geometry, critical for improving interpretability and robustness. While mode connectivity, a lens for analyzing geometric properties of loss landscapes has proven insightful for other deep learning architectures, its implications for GNNs remain unexplored. This work presents the first investigation of mode connectivity in GNNs. We uncover that GNNs exhibit distinct non-linear mode connectivity, diverging from patterns observed in fully-connected networks or CNNs. Crucially, we demonstrate that graph structure, rather than model architecture, dominates this behavior, with graph properties like homophily correlating with mode connectivity patterns. We further establish a link between mode connectivity and generalization, proposing a generalization bound based on loss barriers and revealing its utility as a diagnostic tool. Our findings further bridge theoretical insights with practical implications: they rationalize domain alignment strategies in graph learning and provide a foundation for refining GNN training paradigms.