Do Neural Scaling Laws Exist on Graph Self-Supervised Learning?

TL;DR

This paper investigates whether current graph self-supervised learning methods follow neural scaling laws, finding that performance fluctuations are more influenced by architecture and task design than scale, challenging assumptions for building Graph Foundation Models.

Contribution

The study provides a comprehensive benchmark and analysis showing existing graph SSL techniques do not follow neural scaling laws, highlighting the importance of architecture and task design over scale.

Findings

Graph SSL performance does not consistently improve with scale.

Model architecture and pretext task choice are key performance factors.

Current SSL methods show performance fluctuations rather than scaling behavior.

Abstract

Self-supervised learning~(SSL) is essential to obtain foundation models in NLP and CV domains via effectively leveraging knowledge in large-scale unlabeled data. The reason for its success is that a suitable SSL design can help the model to follow the neural scaling law, i.e., the performance consistently improves with increasing model and dataset sizes. However, it remains a mystery whether existing SSL in the graph domain can follow the scaling behavior toward building Graph Foundation Models~(GFMs) with large-scale pre-training. In this study, we examine whether existing graph SSL techniques can follow the neural scaling behavior with the potential to serve as the essential component for GFMs. Our benchmark includes comprehensive SSL technique implementations with analysis conducted on both the conventional SSL setting and many new settings adopted in other domains. Surprisingly, despite the SSL loss continuously decreasing, no existing graph SSL techniques follow the neural scaling behavior on the downstream performance. The model performance only merely fluctuates on different data scales and model scales. Instead of the scales, the key factors influencing the performance are the choices of model architecture and pretext task design. This paper examines existing SSL techniques for the feasibility of Graph SSL techniques in developing GFMs and opens a new direction for graph SSL design with the new evaluation prototype. Our code implementation is available online to ease reproducibility on https://github.com/GraphSSLScaling/GraphSSLScaling.