Benchmarking Graph Neural Networks in Solving Hard Constraint Satisfaction Problems

TL;DR

This paper introduces new hard benchmark problems for evaluating graph neural networks on difficult constraint satisfaction tasks, revealing that classical algorithms currently outperform GNNs.

Contribution

It provides standardized, challenging benchmarks for GNNs on hard problems and offers a fair comparison showing classical methods still excel.

Findings

Classical heuristics outperform GNNs on new benchmarks

Proposed benchmarks are based on random problem instances

Benchmarks are publicly available for future research

Abstract

Graph neural networks (GNNs) are increasingly applied to hard optimization problems, often claiming superiority over classical heuristics. However, such claims risk being unsolid due to a lack of standard benchmarks on truly hard instances. From a statistical physics perspective, we propose new hard benchmarks based on random problems. We provide these benchmarks, along with performance results from both classical heuristics and GNNs. Our fair comparison shows that classical algorithms still outperform GNNs. We discuss the challenges for neural networks in this domain. Future claims of superiority can be made more robust using our benchmarks, available at https://github.com/ArtLabBocconi/RandCSPBench.