When Active Learning Fails, Uncalibrated Out of Distribution Uncertainty Quantification Might Be the Problem

TL;DR

This paper evaluates the effectiveness of various uncertainty estimation and calibration methods in active learning for materials discovery, highlighting challenges in out-of-distribution uncertainty quantification and its impact on reducing data needs.

Contribution

It systematically compares uncertainty calibration methods across different models and tasks, revealing limitations in out-of-distribution uncertainty estimates for active learning.

Findings

Calibrated uncertainties often fail to generalize out-of-domain.

Uncalibrated and random sampling sometimes outperform calibrated methods in out-of-distribution scenarios.

Intrinsic data and model limitations affect uncertainty quality regardless of model capacity.

Abstract

Efficiently and meaningfully estimating prediction uncertainty is important for exploration in active learning campaigns in materials discovery, where samples with high uncertainty are interpreted as containing information missing from the model. In this work, the effect of different uncertainty estimation and calibration methods are evaluated for active learning when using ensembles of ALIGNN, eXtreme Gradient Boost, Random Forest, and Neural Network model architectures. We compare uncertainty estimates from ALIGNN deep ensembles to loss landscape uncertainty estimates obtained for solubility, bandgap, and formation energy prediction tasks. We then evaluate how the quality of the uncertainty estimate impacts an active learning campaign that seeks model generalization to out-of-distribution data. Uncertainty calibration methods were found to variably generalize from in-domain data to out-of-domain data. Furthermore, calibrated uncertainties were generally unsuccessful in reducing the amount of data required by a model to improve during an active learning campaign on out-of-distribution data when compared to random sampling and uncalibrated uncertainties. The impact of poor-quality uncertainty persists for random forest and eXtreme Gradient Boosting models trained on the same data for the same tasks, indicating that this is at least partially intrinsic to the data and not due to model capacity alone. Analysis of the target, in-distribution uncertainty, out-of-distribution uncertainty, and training residual distributions suggest that future work focus on understanding empirical uncertainties in the feature input space for cases where ensemble prediction variances do not accurately capture the missing information required for the model to generalize.