Confidence Calibration for Systems with Cascaded Predictive Modules

TL;DR

This paper introduces conformal prediction methods for cascaded systems, enabling reliable system-level uncertainty quantification by leveraging module-level data, with theoretical and empirical validation on synthetic and real-world tasks.

Contribution

It presents novel conformal prediction techniques for cascaded modules, addressing the challenge of propagating uncertainty and calibrating system-level prediction intervals without end-to-end data.

Findings

Improved prediction intervals with better system-level guarantees.

Effective calibration using module-level validation data.

Validated on synthetic and real-world indoor navigation data.

Abstract

Existing conformal prediction algorithms estimate prediction intervals at target confidence levels to characterize the performance of a regression model on new test samples. However, considering an autonomous system consisting of multiple modules, prediction intervals constructed for individual modules fall short of accommodating uncertainty propagation over different modules and thus cannot provide reliable predictions on system behavior. We address this limitation and present novel solutions based on conformal prediction to provide prediction intervals calibrated for a predictive system consisting of cascaded modules (e.g., an upstream feature extraction module and a downstream regression module). Our key idea is to leverage module-level validation data to characterize the system-level error distribution without direct access to end-to-end validation data. We provide theoretical justification and empirical experimental results to demonstrate the effectiveness of proposed solutions. In comparison to prediction intervals calibrated for individual modules, our solutions generate improved intervals with more accurate performance guarantees for system predictions, which are demonstrated on both synthetic systems and real-world systems performing overlap prediction for indoor navigation using the Matterport3D dataset.