Feature Overcorrelation in Deep Graph Neural Networks: A New Perspective

TL;DR

This paper introduces the concept of feature overcorrelation in deep GNNs, providing empirical and theoretical evidence, and proposes a framework called DeCorr to mitigate this issue, enabling deeper and more effective GNNs.

Contribution

It presents a new perspective on GNN performance degradation by identifying feature overcorrelation and introduces DeCorr to reduce redundancy in node representations.

Findings

Feature overcorrelation exists in deep GNNs.

DeCorr effectively reduces feature redundancy.

DeCorr enables training of deeper GNNs.

Abstract

Recent years have witnessed remarkable success achieved by graph neural networks (GNNs) in many real-world applications such as recommendation and drug discovery. Despite the success, oversmoothing has been identified as one of the key issues which limit the performance of deep GNNs. It indicates that the learned node representations are highly indistinguishable due to the stacked aggregators. In this paper, we propose a new perspective to look at the performance degradation of deep GNNs, i.e., feature overcorrelation. Through empirical and theoretical study on this matter, we demonstrate the existence of feature overcorrelation in deeper GNNs and reveal potential reasons leading to this issue. To reduce the feature correlation, we propose a general framework DeCorr which can encourage GNNs to encode less redundant information. Extensive experiments have demonstrated that DeCorr can help enable deeper GNNs and is complementary to existing techniques tackling the oversmoothing issue.