Preventing Representational Rank Collapse in MPNNs by Splitting the Computational Graph

TL;DR

This paper introduces a novel message-passing scheme for MPNNs that prevents rank collapse by splitting the computational graph into multi-relational graphs, leading to more informative node representations.

Contribution

It proposes a new approach to mitigate rank collapse in MPNNs by modifying message-passing with multi-relational graphs, ensuring linearly independent node representations.

Findings

Multi-relational graphs improve node representation diversity.

Operating on directed acyclic graphs satisfies the independence condition.

Experimental results confirm enhanced representation quality.

Abstract

The ability of message-passing neural networks (MPNNs) to fit complex functions over graphs is limited as most graph convolutions amplify the same signal across all feature channels, a phenomenon known as rank collapse, and over-smoothing as a special case. Most approaches to mitigate over-smoothing extend common message-passing schemes, e.g., the graph convolutional network, by utilizing residual connections, gating mechanisms, normalization, or regularization techniques. Our work contrarily proposes to directly tackle the cause of this issue by modifying the message-passing scheme and exchanging different types of messages using multi-relational graphs. We identify a sufficient condition to ensure linearly independent node representations. As one instantion, we show that operating on multiple directed acyclic graphs always satisfies our condition and propose to obtain these by defining a strict partial ordering of the nodes. We conduct comprehensive experiments that confirm the benefits of operating on multi-relational graphs to achieve more informative node representations.