Evaluating Robustness and Uncertainty of Graph Models Under Structural Distributional Shifts

TL;DR

This paper introduces a method to evaluate the robustness of graph models under structural distributional shifts, revealing challenges for existing models and highlighting a trade-off between representation quality and distributional separation.

Contribution

It proposes a novel approach to induce structural distributional shifts in graph data and evaluates their impact on various graph models.

Findings

Structural shifts are challenging for current models.

Simple models often outperform complex ones under shifts.

There is a trade-off between representation quality and distributional separation.

Abstract

In reliable decision-making systems based on machine learning, models have to be robust to distributional shifts or provide the uncertainty of their predictions. In node-level problems of graph learning, distributional shifts can be especially complex since the samples are interdependent. To evaluate the performance of graph models, it is important to test them on diverse and meaningful distributional shifts. However, most graph benchmarks considering distributional shifts for node-level problems focus mainly on node features, while structural properties are also essential for graph problems. In this work, we propose a general approach for inducing diverse distributional shifts based on graph structure. We use this approach to create data splits according to several structural node properties: popularity, locality, and density. In our experiments, we thoroughly evaluate the proposed distributional shifts and show that they can be quite challenging for existing graph models. We also reveal that simple models often outperform more sophisticated methods on the considered structural shifts. Finally, our experiments provide evidence that there is a trade-off between the quality of learned representations for the base classification task under structural distributional shift and the ability to separate the nodes from different distributions using these representations.