Benchmarking GNNs for OOD Materials Property Prediction with Uncertainty Quantification

TL;DR

This paper introduces MatUQ, a comprehensive benchmark for evaluating GNNs on out-of-distribution materials property prediction with uncertainty quantification, highlighting the effectiveness of uncertainty-aware training and diverse model performances.

Contribution

The paper presents a new benchmark framework, a novel splitting strategy, and a unified evaluation protocol for GNNs in OOD materials prediction with uncertainty quantification.

Findings

Uncertainty-aware training reduces prediction errors by 70.6% on average.

No single GNN model outperforms others across all tasks.

New uncertainty metric D-EviU correlates strongly with prediction errors.

Abstract

We present MatUQ, a benchmark framework for evaluating graph neural networks (GNNs) on out-of-distribution (OOD) materials property prediction with uncertainty quantification (UQ). MatUQ comprises 1,375 OOD prediction tasks constructed from six materials datasets using five OFM-based and a newly proposed structure-aware splitting strategy, SOAP-LOCO, which captures local atomic environments more effectively. We evaluate 12 representative GNN models under a unified uncertainty-aware training protocol that combines Monte Carlo Dropout and Deep Evidential Regression (DER), and introduce a novel uncertainty metric, D-EviU, which shows the strongest correlation with prediction errors in most tasks. Our experiments yield two key findings. First, the uncertainty-aware training approach significantly improves model prediction accuracy, reducing errors by an average of 70.6\% across challenging OOD scenarios. Second, the benchmark reveals that no single model dominates universally: earlier models such as SchNet and ALIGNN remain competitive, while newer models like CrystalFramer and SODNet demonstrate superior performance on specific material properties. These results provide practical insights for selecting reliable models under distribution shifts in materials discovery.