Trustworthy Few-Shot Transfer of Medical VLMs through Split Conformal Prediction

TL;DR

This paper introduces a novel transductive split conformal adaptation method for medical vision-language models, enhancing trustworthiness and efficiency in small-sample transfer scenarios while maintaining empirical guarantees.

Contribution

It proposes SCA-T, a transductive adaptation pipeline that improves conformal prediction for medical VLMs, addressing limitations of traditional SCP in transfer learning.

Findings

SCA-T outperforms standard SCP in efficiency and coverage.

The method maintains empirical guarantees across various tasks.

Experiments show improved trustworthiness in medical image classification.

Abstract

Medical vision-language models (VLMs) have demonstrated unprecedented transfer capabilities and are being increasingly adopted for data-efficient image classification. Despite its growing popularity, its reliability aspect remains largely unexplored. This work explores the split conformal prediction (SCP) framework to provide trustworthiness guarantees when transferring such models based on a small labeled calibration set. Despite its potential, the generalist nature of the VLMs' pre-training could negatively affect the properties of the predicted conformal sets for specific tasks. While common practice in transfer learning for discriminative purposes involves an adaptation stage, we observe that deploying such a solution for conformal purposes is suboptimal since adapting the model using the available calibration data breaks the rigid exchangeability assumptions for test data in SCP. To address this issue, we propose transductive split conformal adaptation (SCA-T), a novel pipeline for transfer learning on conformal scenarios, which performs an unsupervised transductive adaptation jointly on calibration and test data. We present comprehensive experiments utilizing medical VLMs across various image modalities, transfer tasks, and non-conformity scores. Our framework offers consistent gains in efficiency and conditional coverage compared to SCP, maintaining the same empirical guarantees.