Broken neural scaling laws in materials science

TL;DR

This paper examines neural scaling laws in materials science, revealing that model performance scales predictably with parameters but not with dataset size, highlighting challenges in data efficiency for predicting dielectric functions.

Contribution

It demonstrates that neural scaling laws break down with dataset size but follow a simple power law with model parameters in materials science tasks.

Findings

Scaling with dataset size is broken in this context.

Model performance scales with parameters following a power law.

Dataset size does not improve model accuracy as expected.

Abstract

In materials science, data are scarce and expensive to generate, whether computationally or experimentally. Therefore, it is crucial to identify how model performance scales with dataset size and model capacity to distinguish between data- and model-limited regimes. Neural scaling laws provide a framework for quantifying this behavior and guide the design of materials datasets and machine learning architectures. Here, we investigate neural scaling laws for a paradigmatic materials science task: predicting the dielectric function of metals, a high-dimensional response that governs how solids interact with light. Using over 200,000 dielectric functions from high-throughput ab initio calculations, we study two multi-objective graph neural networks trained to predict the frequency-dependent complex interband dielectric function and the Drude frequency. We observe broken neural scaling laws with respect to dataset size, whereas scaling with the number of model parameters follows a simple power law that rapidly saturates.