Invariant-Based Diagnostics for Graph Benchmarks

TL;DR

This paper introduces graph invariants as a diagnostic tool to evaluate whether graph models truly learn from structural information, revealing that simple invariant-based models often match complex trained models in performance.

Contribution

It proposes using permutation-invariant graph invariants as a diagnostic framework, demonstrating their expressiveness and effectiveness across multiple datasets.

Findings

Invariants outperform standard GNNs in structural characterization.

Invariant-based models are competitive with transformer and message-passing models.

Structural information is often captured by simple, non-trainable invariants.

Abstract

Progress on graph foundation models is hindered by benchmark practices that conflate the contributions of node features and graph structure, making it hard to tell whether a model actually learns from connectivity, or whether it even needs to. We propose addressing this using graph invariants, i.e., permutation-invariant, task-agnostic structural descriptors that serve as a diagnostic framework for graph benchmarks. We show that (i) invariants are more expressive than standard GNNs, (ii) invariants characterize structural heterogeneity within and across benchmark datasets, (iii) invariants predict multi-task performance, and (iv) simple invariant-based models are competitive with, and sometimes exceed, transformer and message-passing baselines across 26 datasets. Our results suggest that expressivity is not the main driver of predictive performance, and that on tasks where structure matters, a non-trainable structural proxy often matches trained message-passing models. We thus posit that invariant baselines should become a standard for evaluating whether structure is required for a task and whether a model picks up on it, serving as a stepping stone towards graph foundation models.