Full Conformal Adaptation of Medical Vision-Language Models

TL;DR

This paper introduces a novel full conformal adaptation method for medical vision-language models, improving their reliability and efficiency in medical image analysis by jointly adapting and conformalizing models with minimal additional data.

Contribution

It proposes a new full conformal adaptation framework and a training-free linear probe, enhancing the reliability of medical VLMs under conformal prediction with minimal data.

Findings

Up to 27% improvement in set efficiency

Maintains coverage guarantees

Effective across multiple medical VLMs and tasks

Abstract

Vision-language models (VLMs) pre-trained at large scale have shown unprecedented transferability capabilities and are being progressively integrated into medical image analysis. Although its discriminative potential has been widely explored, its reliability aspect remains overlooked. This work investigates their behavior under the increasingly popular split conformal prediction (SCP) framework, which theoretically guarantees a given error level on output sets by leveraging a labeled calibration set. However, the zero-shot performance of VLMs is inherently limited, and common practice involves few-shot transfer learning pipelines, which cannot absorb the rigid exchangeability assumptions of SCP. To alleviate this issue, we propose full conformal adaptation, a novel setting for jointly adapting and conformalizing pre-trained foundation models, which operates transductively over each test data point using a few-shot adaptation set. Moreover, we complement this framework with SS-Text, a novel training-free linear probe solver for VLMs that alleviates the computational cost of such a transductive approach. We provide comprehensive experiments using 3 different modality-specialized medical VLMs and 9 adaptation tasks. Our framework requires exactly the same data as SCP, and provides consistent relative improvements of up to 27% on set efficiency while maintaining the same coverage guarantees.