Discovering the Representation Bottleneck of Graph Neural Networks

TL;DR

This paper identifies a fundamental limitation in GNNs called the representation bottleneck, caused by existing graph construction biases, and proposes a dynamic graph rewiring method to improve their ability to capture complex node interactions.

Contribution

The paper reveals the representation bottleneck in GNNs and introduces a novel, interaction pattern-based graph rewiring approach to overcome this limitation.

Findings

Reveals the existence of a representation bottleneck in GNNs.

Demonstrates that graph construction biases contribute to this bottleneck.

Shows that the proposed rewiring method improves GNN performance on various datasets.

Abstract

Graph neural networks (GNNs) rely mainly on the message-passing paradigm to propagate node features and build interactions, and different graph learning problems require different ranges of node interactions. In this work, we explore the capacity of GNNs to capture node interactions under contexts of different complexities. We discover that GNNs usually fail to capture the most informative kinds of interaction styles for diverse graph learning tasks, and thus name this phenomenon GNNs' representation bottleneck. As a response, we demonstrate that the inductive bias introduced by existing graph construction mechanisms can result in this representation bottleneck, \emph{i.e.}, preventing GNNs from learning interactions of the most appropriate complexity. To address that limitation, we propose a novel graph rewiring approach based on interaction patterns learned by GNNs to dynamically adjust each node's receptive fields. Extensive experiments on both real-world and synthetic datasets prove the effectiveness of our algorithm in alleviating the representation bottleneck and its superiority in enhancing the performance of GNNs over state-of-the-art graph rewiring baselines.