Deep Confidence: A Computationally Efficient Framework for Calculating Reliable Errors for Deep Neural Networks

TL;DR

Deep Confidence introduces an efficient framework combining Snapshot Ensembling and conformal prediction to generate reliable confidence intervals for deep neural network predictions, enhancing trust and interpretability in drug discovery applications.

Contribution

It presents a novel, computationally efficient method to estimate prediction confidence intervals for deep learning models using snapshot ensembles and conformal prediction.

Findings

Performs comparably to Random Forest and independent DNN ensembles.

Produces narrower confidence regions than existing methods.

Applicable to diverse datasets with no extra computational cost.

Abstract

Deep learning architectures have proved versatile in a number of drug discovery applications, including the modelling of in vitro compound activity. While controlling for prediction confidence is essential to increase the trust, interpretability and usefulness of virtual screening models in drug discovery, techniques to estimate the reliability of the predictions generated with deep learning networks remain largely underexplored. Here, we present Deep Confidence, a framework to compute valid and efficient confidence intervals for individual predictions using the deep learning technique Snapshot Ensembling and conformal prediction. Specifically, Deep Confidence generates an ensemble of deep neural networks by recording the network parameters throughout the local minima visited during the optimization phase of a single neural network. This approach serves to derive a set of base learners (i.e., snapshots) with comparable predictive power on average, that will however generate slightly different predictions for a given instance. The variability across base learners and the validation residuals are in turn harnessed to compute confidence intervals using the conformal prediction framework. Using a set of 24 diverse IC50 data sets from ChEMBL 23, we show that Snapshot Ensembles perform on par with Random Forest (RF) and ensembles of independently trained deep neural networks. In addition, we find that the confidence regions predicted using the Deep Confidence framework span a narrower set of values. Overall, Deep Confidence represents a highly versatile error prediction framework that can be applied to any deep learning-based application at no extra computational cost.