Doubly Debiased Test-Time Prompt Tuning for Vision-Language Models

TL;DR

This paper introduces a novel test-time prompt tuning method for vision-language models that reduces bias by leveraging dynamic knowledge retrieval and confidence-based regularization, improving generalization across diverse datasets.

Contribution

The paper proposes a doubly debiased prompt tuning approach combining knowledge retrieval and confidence-aware regularization to mitigate prompt optimization bias in vision-language models.

Findings

Outperforms baselines on 15 benchmark datasets.

Effectively reduces prompt optimization bias.

Enhances generalization under distribution shifts.

Abstract

Test-time prompt tuning for vision-language models has demonstrated impressive generalization capabilities under zero-shot settings. However, tuning the learnable prompts solely based on unlabeled test data may induce prompt optimization bias, ultimately leading to suboptimal performance on downstream tasks. In this work, we analyze the underlying causes of prompt optimization bias from both the model and data perspectives. In terms of the model, the entropy minimization objective typically focuses on reducing the entropy of model predictions while overlooking their correctness. This can result in overconfident yet incorrect outputs, thereby compromising the quality of prompt optimization. On the data side, prompts affected by optimization bias can introduce misalignment between visual and textual modalities, which further aggravates the prompt optimization bias. To this end, we propose a Doubly Debiased Test-Time Prompt Tuning method. Specifically, we first introduce a dynamic retrieval-augmented modulation module that retrieves high-confidence knowledge from a dynamic knowledge base using the test image feature as a query, and uses the retrieved knowledge to modulate the predictions. Guided by the refined predictions, we further develop a reliability-aware prompt optimization module that incorporates a confidence-based weighted ensemble and cross-modal consistency distillation to impose regularization constraints during prompt tuning. Extensive experiments across 15 benchmark datasets involving both natural distribution shifts and cross-datasets generalization demonstrate that our method outperforms baselines, validating its effectiveness in mitigating prompt optimization bias.