Self-supervised Learning on Graphs: Deep Insights and New Direction

TL;DR

This paper investigates the application of self-supervised learning to graph neural networks, providing deep insights into when and why SSL works for graphs, and proposing a new method called SelfTask that achieves state-of-the-art results.

Contribution

It offers a comprehensive empirical analysis of SSL strategies on graphs and introduces SelfTask, a novel approach for designing effective pretext tasks for GNNs.

Findings

Empirical study reveals which SSL strategies are most effective for GNNs.

SelfTask outperforms existing SSL methods on multiple real-world datasets.

Deep insights guide the design of advanced pretext tasks for graph SSL.

Abstract

The success of deep learning notoriously requires larger amounts of costly annotated data. This has led to the development of self-supervised learning (SSL) that aims to alleviate this limitation by creating domain specific pretext tasks on unlabeled data. Simultaneously, there are increasing interests in generalizing deep learning to the graph domain in the form of graph neural networks (GNNs). GNNs can naturally utilize unlabeled nodes through the simple neighborhood aggregation that is unable to thoroughly make use of unlabeled nodes. Thus, we seek to harness SSL for GNNs to fully exploit the unlabeled data. Different from data instances in the image and text domains, nodes in graphs present unique structure information and they are inherently linked indicating not independent and identically distributed (or i.i.d.). Such complexity is a double-edged sword for SSL on graphs. On the one hand, it determines that it is challenging to adopt solutions from the image and text domains to graphs and dedicated efforts are desired. On the other hand, it provides rich information that enables us to build SSL from a variety of perspectives. Thus, in this paper, we first deepen our understandings on when, why, and which strategies of SSL work with GNNs by empirically studying numerous basic SSL pretext tasks on graphs. Inspired by deep insights from the empirical studies, we propose a new direction SelfTask to build advanced pretext tasks that are able to achieve state-of-the-art performance on various real-world datasets. The specific experimental settings to reproduce our results can be found in \url{https://github.com/ChandlerBang/SelfTask-GNN}.