How (Mis)calibrated is Your Federated CLIP and What To Do About It?

TL;DR

This paper investigates the calibration issues of federated CLIP models, analyzes existing methods, and proposes FL2oRA, a LoRA-based approach that improves calibration and reliability in federated learning settings.

Contribution

It introduces FL2oRA, a simple LoRA-based method that enhances calibration in federated CLIP models, addressing a key challenge in distributed vision-language model fine-tuning.

Findings

Textual Prompt Tuning degrades calibration under FL

Existing calibration techniques offer limited improvements

FL2oRA consistently produces well-calibrated models

Abstract

While vision-language models like CLIP have been extensively studied, their calibration, crucial for reliable predictions, has received limited attention. Although a few prior works have examined CLIP calibration in offline settings, the impact of fine-tuning CLIP in a federated learning (FL) setup remains unexplored. In this work, we investigate how FL affects CLIP calibration and propose strategies to improve reliability in this distributed setting. We first analyze Textual Prompt Tuning approaches and show that they degrade calibration metrics when operating under FL. We also evaluate existing in-training calibration techniques across four global aggregation methods, finding that they provide limited improvements. Our results suggest that the key challenge lies not only in how we aggregate or calibrate, but in which components we choose to fine-tune. Motivated by this insight, we propose $\text{FL}^2\text{oRA}$, a straightforward LoRA-based approach that naturally improves calibration in FL, and we analyze the factors behind its effectiveness. Experiments on multiple benchmarks demonstrate that $\text{FL}^2\text{oRA}$ consistently produces well-calibrated models, reducing the need for explicit calibration procedures. Codes are available at https://github.com/mainaksingha01/FL2oRA.