Conformal Prediction for Multimodal Regression

TL;DR

This paper extends conformal prediction to multimodal data by utilizing internal neural network features, enabling distribution-free uncertainty quantification in complex, multimodal scenarios.

Contribution

The paper introduces a novel methodology for applying conformal prediction to multimodal data using neural network internal features.

Findings

Internal neural network features can be used to construct prediction intervals.

The approach enables conformal prediction in multimodal contexts.

Potential for broader application in multimodal data domains.

Abstract

This paper introduces multimodal conformal regression. Traditionally confined to scenarios with solely numerical input features, conformal prediction is now extended to multimodal contexts through our methodology, which harnesses internal features from complex neural network architectures processing images and unstructured text. Our findings highlight the potential for internal neural network features, extracted from convergence points where multimodal information is combined, to be used by conformal prediction to construct prediction intervals (PIs). This capability paves new paths for deploying conformal prediction in domains abundant with multimodal data, enabling a broader range of problems to benefit from guaranteed distribution-free uncertainty quantification.