Towards overcoming data scarcity in materials science: unifying models and datasets with a mixture of experts framework

TL;DR

This paper introduces a mixture of experts framework that effectively combines multiple models and datasets to improve property prediction in materials science, especially when data is scarce, outperforming traditional transfer learning methods.

Contribution

The paper presents a novel, scalable, and interpretable mixture of experts approach that automatically integrates information from various sources for materials property prediction.

Findings

Outperforms pairwise transfer learning on 16 of 19 tasks

Automatically learns to combine multiple source models and datasets

Provides an interpretable and scalable transfer mechanism

Abstract

While machine learning has emerged in recent years as a useful tool for rapid prediction of materials properties, generating sufficient data to reliably train models without overfitting is still impractical for many applications. Towards overcoming this limitation, we present a general framework for leveraging complementary information across different models and datasets for accurate prediction of data scarce materials properties. Our approach, based on a machine learning paradigm called mixture of experts, outperforms pairwise transfer learning on 16 of 19 materials property regression tasks, performing comparably on the remaining three. Unlike pairwise transfer learning, our framework automatically learns to combine information from multiple source tasks in a single training run, alleviating the need for brute-force experiments to determine which source task to transfer from. The approach also provides an interpretable, model-agnostic, and scalable mechanism to transfer information from an arbitrary number of models and datasets to any downstream property prediction task. We anticipate the performance of our framework will further improve as better model architectures, new pre-training tasks, and larger materials datasets are developed by the community.