Message-Passing GNNs Fail to Approximate Sparse Triangular Factorizations

TL;DR

This paper shows that message-passing GNNs cannot effectively approximate sparse triangular factorizations, highlighting the need for new architectures to handle non-local dependencies in scientific computing tasks.

Contribution

The paper provides both theoretical and empirical evidence that message-passing GNNs are fundamentally limited in approximating sparse triangular factorizations, especially for matrices requiring non-local dependencies.

Findings

GNNs perform poorly on matrices with non-local dependencies, with cosine similarity below 0.6.

Architectural innovations beyond message-passing are necessary for scientific computing tasks.

Even non-local Graph Transformers fail to match specialized baselines.

Abstract

Graph Neural Networks (GNNs) have been proposed as a tool for learning sparse matrix preconditioners, which are key components in accelerating linear solvers. This position paper argues that message-passing GNNs are fundamentally incapable of approximating sparse triangular factorizations. We demonstrate that message-passing GNNs fundamentally fail to approximate sparse triangular factorizations for classes of matrices for which high-quality preconditioners exist but require non-local dependencies. To illustrate this, we construct a set of baselines using both synthetic matrices and real-world examples from the SuiteSparse collection. Across a range of GNN architectures, including Graph Attention Networks and Graph Transformers, we observe severe performance degradation compared to exact or K-optimal factorizations, with cosine similarity dropping below $0.6$ in key cases. Our theoretical and empirical results suggest that architectural innovations beyond message-passing are necessary for applying GNNs to scientific computing tasks such as matrix factorization. Experiments demonstrate that overcoming non-locality alone is insufficient. Tailored architectures are necessary to capture the required dependencies since even a completely non-local Graph Transformer fails to match the proposed baselines.