On the Global Self-attention Mechanism for Graph Convolutional Networks

TL;DR

This paper investigates the application of Global Self-attention (GSA) in Graph Convolutional Networks (GCNs), demonstrating its ability to enhance expressive power and mitigate overfitting and over-smoothing issues through theoretical analysis and experiments.

Contribution

It introduces the GSA mechanism into GCNs, providing theoretical insights and empirical evidence of improved performance and robustness.

Findings

GSA enables GCNs to capture feature-based relations independently of edges.

GSA alleviates overfitting and over-smoothing in GCNs.

GSA-augmented GCNs outperform baseline models on benchmark datasets.

Abstract

Applying Global Self-attention (GSA) mechanism over features has achieved remarkable success on Convolutional Neural Networks (CNNs). However, it is not clear if Graph Convolutional Networks (GCNs) can similarly benefit from such a technique. In this paper, inspired by the similarity between CNNs and GCNs, we study the impact of the Global Self-attention mechanism on GCNs. We find that consistent with the intuition, the GSA mechanism allows GCNs to capture feature-based vertex relations regardless of edge connections; As a result, the GSA mechanism can introduce extra expressive power to the GCNs. Furthermore, we analyze the impacts of the GSA mechanism on the issues of overfitting and over-smoothing. We prove that the GSA mechanism can alleviate both the overfitting and the over-smoothing issues based on some recent technical developments. Experiments on multiple benchmark datasets illustrate both superior expressive power and less significant overfitting and over-smoothing problems for the GSA-augmented GCNs, which corroborate the intuitions and the theoretical results.