GraphMAE: Self-Supervised Masked Graph Autoencoders

TL;DR

GraphMAE introduces a masked graph autoencoder that focuses on feature reconstruction with a novel masking strategy and scaled cosine error, leading to consistent outperformance in various graph learning tasks.

Contribution

The paper proposes GraphMAE, a generative self-supervised graph autoencoder that improves robustness and performance by shifting from structure to feature reconstruction and employing new training techniques.

Findings

GraphMAE outperforms state-of-the-art contrastive and generative methods.

It demonstrates strong results across 21 datasets and multiple tasks.

The approach enhances robustness and training stability.

Abstract

Self-supervised learning (SSL) has been extensively explored in recent years. Particularly, generative SSL has seen emerging success in natural language processing and other AI fields, such as the wide adoption of BERT and GPT. Despite this, contrastive learning-which heavily relies on structural data augmentation and complicated training strategies-has been the dominant approach in graph SSL, while the progress of generative SSL on graphs, especially graph autoencoders (GAEs), has thus far not reached the potential as promised in other fields. In this paper, we identify and examine the issues that negatively impact the development of GAEs, including their reconstruction objective, training robustness, and error metric. We present a masked graph autoencoder GraphMAE that mitigates these issues for generative self-supervised graph pretraining. Instead of reconstructing graph structures, we propose to focus on feature reconstruction with both a masking strategy and scaled cosine error that benefit the robust training of GraphMAE. We conduct extensive experiments on 21 public datasets for three different graph learning tasks. The results manifest that GraphMAE-a simple graph autoencoder with careful designs-can consistently generate outperformance over both contrastive and generative state-of-the-art baselines. This study provides an understanding of graph autoencoders and demonstrates the potential of generative self-supervised pre-training on graphs.