Contrast-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

TL;DR

This paper introduces Contrast-Aware Calibration (CAC), a novel multimodal calibration method for fine-tuned CLIP models that improves confidence calibration on both train and unseen classes without additional training or data analysis.

Contribution

The paper proposes CAC, a contrastive difference-based calibration method that effectively calibrates both train and unseen classes in fine-tuned CLIP models, addressing limitations of existing methods.

Findings

CAC outperforms existing calibration methods on 11 datasets.

CAC maintains accuracy and inference speed while improving calibration.

CAC effectively calibrates both train and unseen classes in various fine-tuning scenarios.

Abstract

Vision-language models (VLMs), such as CLIP, have demonstrated exceptional generalization capabilities and can quickly adapt to downstream tasks through prompt fine-tuning. Unfortunately, in classification tasks involving non-training classes, known as open-vocabulary setting, fine-tuned VLMs often overfit to train classes, resulting in a misalignment between confidence scores and actual accuracy on unseen classes, which significantly undermines their reliability in real-world deployments. Existing confidence calibration methods typically require training parameters or analyzing features from the training dataset, restricting their ability to generalize unseen classes without corresponding train data. Moreover, VLM-specific calibration methods rely solely on text features from train classes as calibration indicators, which inherently limits their ability to calibrate train classes. To address these challenges, we propose an effective multimodal calibration method Contrast-Aware Calibration (CAC). Building on the original CLIP's zero-shot adaptability and the conclusion from empirical analysis that poor intra-class and inter-class discriminative ability on unseen classes is the root cause, we calculate calibration weights based on the contrastive difference between the original and fine-tuned CLIP. This method not only adapts to calibrating unseen classes but also overcomes the limitations of previous VLM calibration methods that could not calibrate train classes. In experiments involving 11 datasets with 5 fine-tuning methods, CAC consistently achieved the best calibration effect on both train and unseen classes without sacrificing accuracy and inference speed.