Temporal Distribution Shift in Real-World Pharmaceutical Data: Implications for Uncertainty Quantification in QSAR Models

TL;DR

This study evaluates how distribution shifts over time in real-world pharmaceutical data affect the reliability of uncertainty quantification methods in QSAR models, revealing significant challenges in maintaining accuracy under such shifts.

Contribution

It provides a large-scale, systematic evaluation of uncertainty estimation methods in the context of real-world distribution shifts in pharmaceutical data.

Findings

Significant temporal shifts in label and descriptor space.

Distribution shifts impair the performance of uncertainty estimation methods.

Certain assay types exhibit more pronounced shifts.

Abstract

The estimation of uncertainties associated with predictions from quantitative structure-activity relationship (QSAR) models can accelerate the drug discovery process by identifying promising experiments and allowing an efficient allocation of resources. Several computational tools exist that estimate the predictive uncertainty in machine learning models. However, deviations from the i.i.d. setting have been shown to impair the performance of these uncertainty quantification methods. We use a real-world pharmaceutical dataset to address the pressing need for a comprehensive, large-scale evaluation of uncertainty estimation methods in the context of realistic distribution shifts over time. We investigate the performance of several uncertainty estimation methods, including ensemble-based and Bayesian approaches. Furthermore, we use this real-world setting to systematically assess the distribution shifts in label and descriptor space and their impact on the capability of the uncertainty estimation methods. Our study reveals significant shifts over time in both label and descriptor space and a clear connection between the magnitude of the shift and the nature of the assay. Moreover, we show that pronounced distribution shifts impair the performance of popular uncertainty estimation methods used in QSAR models. This work highlights the challenges of identifying uncertainty quantification methods that remain reliable under distribution shifts introduced by real-world data.