Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration

TL;DR

This paper introduces Conformal Predictive Self-Calibration (CPSC), a unified framework for improving multimodal learning with low-quality data by on-the-fly self-calibration of features and gradients using conformal prediction.

Contribution

The paper proposes a novel CPSC framework that integrates representation and gradient self-calibration with a self-updating conformal predictor for robust multimodal learning.

Findings

CPSC outperforms state-of-the-art methods on six benchmark datasets.

The framework effectively handles modality imbalance and noisy corruption.

Self-calibration improves feature resilience and training reliability.

Abstract

Multimodal learning often grapples with the challenge of low-quality data, which predominantly manifests as two facets: modality imbalance and noisy corruption. While these issues are often studied in isolation, we argue that they share a common root in the predictive uncertainty towards the reliability of individual modalities and instances during learning. In this paper, we propose a unified framework, termed Conformal Predictive Self-Calibration (CPSC), which leverages conformal prediction to equip the model with the ability to perform self-guided calibration on-the-fly. The core of our proposed CPSC lies in a novel self-calibrating training loop that seamlessly integrates two key modules: (1) Representation Self-Calibration, which decomposes unimodal features into components, and selectively fuses the most robust ones identified by a conformal predictor to enhance feature resilience. (2) Gradient Self-Calibration, which recalibrates the gradient flow during backpropagation based on instance-wise reliability scores, steering the optimization towards more trustworthy directions. Furthermore, we also devise a self-update strategy for the conformal predictor to ensure the entire system co-evolves consistently throughout the training process. Extensive experiments on six benchmark datasets under both imbalanced and noisy settings demonstrate that our CPSC framework consistently outperforms existing state-of-the-art methods. Our code is available at https://github.com/XunCHN/CPSC.