Tackling Over-Smoothing for General Graph Convolutional Networks

TL;DR

This paper provides a theoretical analysis of over-smoothing in deep GCNs, introduces DropEdge as a method to mitigate it, and demonstrates its effectiveness through experiments on various datasets.

Contribution

It offers the first theoretical analysis of over-smoothing in general GCNs and proposes DropEdge to alleviate this issue, improving deep GCN performance.

Findings

All GCN variants converge to a cuboid, causing over-smoothing.

DropEdge reduces convergence speed of over-smoothing.

DropEdge improves performance on multiple benchmarks.

Abstract

Increasing the depth of GCN, which is expected to permit more expressivity, is shown to incur performance detriment especially on node classification. The main cause of this lies in over-smoothing. The over-smoothing issue drives the output of GCN towards a space that contains limited distinguished information among nodes, leading to poor expressivity. Several works on refining the architecture of deep GCN have been proposed, but it is still unknown in theory whether or not these refinements are able to relieve over-smoothing. In this paper, we first theoretically analyze how general GCNs act with the increase in depth, including generic GCN, GCN with bias, ResGCN, and APPNP. We find that all these models are characterized by a universal process: all nodes converging to a cuboid. Upon this theorem, we propose DropEdge to alleviate over-smoothing by randomly removing a certain number of edges at each training epoch. Theoretically, DropEdge either reduces the convergence speed of over-smoothing or relieves the information loss caused by dimension collapse. Experimental evaluations on simulated dataset have visualized the difference in over-smoothing between different GCNs. Moreover, extensive experiments on several real benchmarks support that DropEdge consistently improves the performance on a variety of both shallow and deep GCNs.