Achieving Well-Informed Decision-Making in Drug Discovery: A Comprehensive Calibration Study using Neural Network-Based Structure-Activity Models

TL;DR

This paper evaluates and improves the calibration of neural network models predicting drug-target interactions, introducing Bayesian Linear Probing to enhance uncertainty estimates and combining calibration techniques for better decision-making in drug discovery.

Contribution

It proposes Bayesian Linear Probing for efficient Bayesian uncertainty estimation and demonstrates how combining calibration methods improves model reliability in drug discovery.

Findings

BLP enhances model calibration and uncertainty estimation.

Combining calibration methods improves predictive accuracy and reliability.

Well-calibrated models facilitate better decision-making in drug discovery.

Abstract

In the drug discovery process, where experiments can be costly and time-consuming, computational models that predict drug-target interactions are valuable tools to accelerate the development of new therapeutic agents. Estimating the uncertainty inherent in these neural network predictions provides valuable information that facilitates optimal decision-making when risk assessment is crucial. However, such models can be poorly calibrated, which results in unreliable uncertainty estimates that do not reflect the true predictive uncertainty. In this study, we compare different metrics, including accuracy and calibration scores, used for model hyperparameter tuning to investigate which model selection strategy achieves well-calibrated models. Furthermore, we propose to use a computationally efficient Bayesian uncertainty estimation method named Bayesian Linear Probing (BLP), which generates Hamiltonian Monte Carlo (HMC) trajectories to obtain samples for the parameters of a Bayesian Logistic Regression fitted to the hidden layer of the baseline neural network. We report that BLP improves model calibration and achieves the performance of common uncertainty quantification methods by combining the benefits of uncertainty estimation and probability calibration methods. Finally, we show that combining post hoc calibration method with well-performing uncertainty quantification approaches can boost model accuracy and calibration.