Re-evaluating the Advancements of Heterophilic Graph Learning

TL;DR

This paper critically re-evaluates heterophilic graph learning by fine-tuning models on challenging datasets, proposing a new heterophily taxonomy, and quantitatively assessing homophily metrics to improve evaluation standards.

Contribution

It introduces a new heterophily taxonomy, thoroughly re-evaluates SOTA GNNs with fine-tuned hyperparameters, and provides the first quantitative assessment of homophily metrics on synthetic graphs.

Findings

Malignant and ambiguous heterophily are the truly challenging datasets.

Fine-tuned hyperparameters significantly impact GNN performance.

Quantitative evaluation reveals limitations of existing homophily metrics.

Abstract

Over the past decade, Graph Neural Networks (GNNs) have achieved great success on machine learning tasks with relational data. However, recent studies have found that heterophily can cause significant performance degradation of GNNs, especially on node-level tasks. Numerous heterophilic benchmark datasets have been put forward to validate the efficacy of heterophily-specific GNNs, and various homophily metrics have been designed to help recognize these challenging datasets. Nevertheless, there still exist multiple pitfalls that severely hinder the proper evaluation of new models and metrics: 1) lack of hyperparameter tuning; 2) insufficient evaluation on the truly challenging heterophilic datasets; 3) missing quantitative evaluation for homophily metrics on synthetic graphs. To overcome these challenges, we first train and fine-tune baseline models on $27$ most widely used benchmark datasets, and categorize them into three distinct groups: malignant, benign and ambiguous heterophilic datasets. We identify malignant and ambiguous heterophily as the truly challenging subsets of tasks, and to our best knowledge, we are the first to propose such taxonomy. Then, we re-evaluate $11$ state-of-the-arts (SOTA) GNNs, covering six popular methods, with fine-tuned hyperparameters on different groups of heterophilic datasets. Based on the model performance, we comprehensively reassess the effectiveness of different methods on heterophily. At last, we evaluate $11$ popular homophily metrics on synthetic graphs with three different graph generation approaches. To overcome the unreliability of observation-based comparison and evaluation, we conduct the first quantitative evaluation and provide detailed analysis.