Development and Evaluation of Conformal Prediction Methods for QSAR

TL;DR

This paper develops and tests conformal prediction methods tailored for QSAR models, providing valid uncertainty estimates for biological activity predictions across various datasets and advanced machine learning techniques.

Contribution

It introduces computationally efficient conformal prediction algorithms specifically designed for modern ML models in QSAR, enhancing uncertainty quantification.

Findings

Conformal predictors are valid across diverse QSAR datasets.

Proposed methods are computationally efficient for deep learning models.

Prediction intervals achieve desired coverage probabilities.

Abstract

The quantitative structure-activity relationship (QSAR) regression model is a commonly used technique for predicting biological activities of compounds using their molecular descriptors. Predictions from QSAR models can help, for example, to optimize molecular structure; prioritize compounds for further experimental testing; and estimate their toxicity. In addition to the accurate estimation of the activity, it is highly desirable to obtain some estimate of the uncertainty associated with the prediction, e.g., calculate a prediction interval (PI) containing the true molecular activity with a pre-specified probability, say 70%, 90% or 95%. The challenge is that most machine learning (ML) algorithms that achieve superior predictive performance require some add-on methods for estimating uncertainty of their prediction. The development of these algorithms is an active area of research by statistical and ML communities but their implementation for QSAR modeling remains limited. Conformal prediction (CP) is a promising approach. It is agnostic to the prediction algorithm and can produce valid prediction intervals under some weak assumptions on the data distribution. We proposed computationally efficient CP algorithms tailored to the most advanced ML models, including Deep Neural Networks and Gradient Boosting Machines. The validity and efficiency of proposed conformal predictors are demonstrated on a diverse collection of QSAR datasets as well as simulation studies.