Evaluating Point-Prediction Uncertainties in Neural Networks for Drug Discovery

TL;DR

This paper evaluates methods for quantifying different sources of uncertainty in neural network predictions to improve drug discovery processes, emphasizing the importance of understanding uncertainty sources for better decision-making.

Contribution

It introduces a framework for separately modeling various types of predictive uncertainty in neural networks used for drug discovery, utilizing chemical compound partitions for evaluation.

Findings

Uncertainty estimates vary with data partitions and featurization schemes.

Different UQ methods capture distinct sources of uncertainty.

Uncertainty measures correlate with prediction errors.

Abstract

Neural Network (NN) models provide potential to speed up the drug discovery process and reduce its failure rates. The success of NN models require uncertainty quantification (UQ) as drug discovery explores chemical space beyond the training data distribution. Standard NN models do not provide uncertainty information. Methods that combine Bayesian models with NN models address this issue, but are difficult to implement and more expensive to train. Some methods require changing the NN architecture or training procedure, limiting the selection of NN models. Moreover, predictive uncertainty can come from different sources. It is important to have the ability to separately model different types of predictive uncertainty, as the model can take assorted actions depending on the source of uncertainty. In this paper, we examine UQ methods that estimate different sources of predictive uncertainty for NN models aiming at drug discovery. We use our prior knowledge on chemical compounds to design the experiments. By utilizing a visualization method we create non-overlapping and chemically diverse partitions from a collection of chemical compounds. These partitions are used as training and test set splits to explore NN model uncertainty. We demonstrate how the uncertainties estimated by the selected methods describe different sources of uncertainty under different partitions and featurization schemes and the relationship to prediction error.