Technical report on label-informed logit redistribution for better domain generalization in low-shot classification with foundation models

TL;DR

This paper introduces a confidence misalignment penalty (CMP) for fine-tuning foundation models, significantly improving calibration in low-shot vision classification and domain generalization tasks.

Contribution

It proposes a novel penalty method that enhances confidence calibration by adjusting logit scores during fine-tuning of foundation models.

Findings

CMP improves Expected Calibration Error (ECE) by up to 9.72%.

The method outperforms existing prompt learning techniques.

Experiments on 12 vision and 5 domain datasets validate effectiveness.

Abstract

Confidence calibration is an emerging challenge in real-world decision systems based on foundations models when used for downstream vision classification tasks. Due to various reasons exposed, logit scores on the CLIP head remain large irrespective of whether the image-language pairs reconcile. It is difficult to address in data space, given the few-shot regime. We propose a penalty incorporated into loss objective that penalizes incorrect classifications whenever one is made during finetuning, by moving an amount of log-likelihood to the true class commensurate to the relative amplitudes of the two likelihoods. We refer to it as \textit{confidence misalignment penalty (CMP)}. Extensive experiments on $12$ vision datasets and $5$ domain generalization datasets supports the calibration performance of our method against stat-of-the-art. CMP outperforms the benchmarked prompt learning methods, demonstrating average improvement in Expected Calibration Error (ECE) by average $6.01$\%, $4.01$ \% at minimum and $9.72$\% at maximum.