Fair Context Learning for Evidence-Balanced Test-Time Adaptation in Vision-Language Models

TL;DR

This paper introduces Fair Context Learning (FCL), a novel test-time adaptation framework for vision-language models that improves robustness under distribution shifts by avoiding entropy minimization and addressing shared-evidence bias.

Contribution

FCL is a new episodic TTA method that decouples adaptation into exploration and fairness-driven calibration, enhancing robustness without entropy minimization.

Findings

FCL achieves competitive performance on diverse benchmarks.

It effectively mitigates overconfidence and spurious correlations.

The approach improves calibration of text embeddings.

Abstract

Vision-Language Models (VLMs) such as CLIP enable strong zero-shot recognition but suffer substantial degradation under distribution shifts. Test-Time Adaptation (TTA) aims to improve robustness using only unlabeled test samples, yet most prompt-based TTA methods rely on entropy minimization -- an approach that can amplify spurious correlations and induce overconfident errors when classes share visual features. We propose Fair Context Learning (FCL), an episodic TTA framework that avoids entropy minimization by explicitly addressing shared-evidence bias. Motivated by our additive evidence decomposition assumption, FCL decouples adaptation into (i) augmentation-based exploration to identify plausible class candidates, and (ii) fairness-driven calibration that adapts text contexts to equalize sensitivity to common visual evidence. This fairness constraint mitigates partial feature obsession and enables effective calibration of text embeddings without relying on entropy reduction. Through extensive evaluation, we empirically validate our theoretical motivation and show that FCL achieves competitive adaptation performance relative to state-of-the-art TTA methods across diverse domain-shift and fine-grained benchmarks.