Revisiting the Necessity of Graph Learning and Common Graph Benchmarks

TL;DR

This paper questions the necessity of graph structure in many benchmarks, showing that node features often suffice for high performance and proposing synthetic datasets to better evaluate graph learning methods.

Contribution

The authors demonstrate that many standard graph benchmarks can be solved with node features alone, challenging the assumption that graph structure is always essential, and introduce synthetic datasets for more meaningful evaluation.

Findings

Node features often outperform graph structure in common benchmarks.

Graph structure provides limited benefit on five out of seven datasets tested.

Synthetic datasets are proposed to better assess the true capabilities of graph neural networks.

Abstract

Graph machine learning has enjoyed a meteoric rise in popularity since the introduction of deep learning in graph contexts. This is no surprise due to the ubiquity of graph data in large scale industrial settings. Tacitly assumed in all graph learning tasks is the separation of the graph structure and node features: node features strictly encode individual data while the graph structure consists only of pairwise interactions. The driving belief is that node features are (by themselves) insufficient for these tasks, so benchmark performance accurately reflects improvements in graph learning. In our paper, we challenge this orthodoxy by showing that, surprisingly, node features are oftentimes more-than-sufficient for many common graph benchmarks, breaking this critical assumption. When comparing against a well-tuned feature-only MLP baseline on seven of the most commonly used graph learning datasets, one gains little benefit from using graph structure on five datasets. We posit that these datasets do not benefit considerably from graph learning because the features themselves already contain enough graph information to obviate or substantially reduce the need for the graph. To illustrate this point, we perform a feature study on these datasets and show how the features are responsible for closing the gap between MLP and graph-method performance. Further, in service of introducing better empirical measures of progress for graph neural networks, we present a challenging parametric family of principled synthetic datasets that necessitate graph information for nontrivial performance. Lastly, we section out a subset of real-world datasets that are not trivially solved by an MLP and hence serve as reasonable benchmarks for graph neural networks.