Distribution-free risk assessment of regression-based machine learning algorithms

TL;DR

This paper introduces a conformal prediction-based method for risk assessment in regression machine learning models, providing conservative failure probability estimates with guarantees, validated through extensive experiments under various conditions.

Contribution

The paper develops a novel conformal prediction approach for risk assessment in regression, ensuring conservative failure probability estimates with theoretical guarantees.

Findings

The method provides conservative failure probability estimates.

Empirical validation shows robustness under covariate shift.

Performance varies with dataset size and modeling regime.

Abstract

Machine learning algorithms have grown in sophistication over the years and are increasingly deployed for real-life applications. However, when using machine learning techniques in practical settings, particularly in high-risk applications such as medicine and engineering, obtaining the failure probability of the predictive model is critical. We refer to this problem as the risk-assessment task. We focus on regression algorithms and the risk-assessment task of computing the probability of the true label lying inside an interval defined around the model's prediction. We solve the risk-assessment problem using the conformal prediction approach, which provides prediction intervals that are guaranteed to contain the true label with a given probability. Using this coverage property, we prove that our approximated failure probability is conservative in the sense that it is not lower than the true failure probability of the ML algorithm. We conduct extensive experiments to empirically study the accuracy of the proposed method for problems with and without covariate shift. Our analysis focuses on different modeling regimes, dataset sizes, and conformal prediction methodologies.