Data efficiency and extrapolation trends in neural network interatomic potentials

TL;DR

This paper investigates how architectural and optimization choices affect the generalization, stability, and data efficiency of neural network interatomic potentials, introducing loss landscape visualization and loss entropy as predictive tools.

Contribution

It introduces loss entropy as a metric to predict out-of-distribution error and MD stability, guiding the development of more robust NNIP architectures.

Findings

Loss entropy predicts out-of-distribution error.

Architectural choices influence extrapolation behavior.

Flatter loss landscapes correlate with better data efficiency.

Abstract

Over the last few years, key architectural advances have been proposed for neural network interatomic potentials (NNIPs), such as incorporating message-passing networks, equivariance, or many-body expansion terms. Although modern NNIP models exhibit small differences in energy/forces errors, improvements in accuracy are still considered the main target when developing new NNIP architectures. In this work, we show how architectural and optimization choices influence the generalization of NNIPs, revealing trends in molecular dynamics (MD) stability, data efficiency, and loss landscapes. Using the 3BPA dataset, we show that test errors in NNIP follow a scaling relation and can be robust to noise, but cannot predict MD stability in the high-accuracy regime. To circumvent this problem, we propose the use of loss landscape visualizations and a metric of loss entropy for predicting the generalization power of NNIPs. With a large-scale study on NequIP and MACE, we show that the loss entropy predicts out-of-distribution error and MD stability despite being computed only on the training set. Using this probe, we demonstrate how the choice of optimizers, loss function weighting, data normalization, and other architectural decisions influence the extrapolation behavior of NNIPs. Finally, we relate loss entropy to data efficiency, demonstrating that flatter landscapes also predict learning curve slopes. Our work provides a deep learning justification for the extrapolation performance of many common NNIPs, and introduces tools beyond accuracy metrics that can be used to inform the development of next-generation models.