Prototype-Guided Pseudo-Labeling with Neighborhood-Aware Consistency for Unsupervised Adaptation

TL;DR

This paper introduces a novel adaptive pseudo-labeling framework for vision-language models like CLIP, improving unsupervised domain adaptation by leveraging prototype and neighborhood consistency to generate more accurate pseudo-labels.

Contribution

The work presents a new pseudo-labeling method combining prototype and neighborhood consistency, with an adaptive weighting mechanism, advancing unsupervised adaptation performance.

Findings

Achieves state-of-the-art results on 11 benchmarks.

Produces more accurate pseudo-labels under domain shifts.

Maintains computational efficiency during adaptation.

Abstract

In unsupervised adaptation for vision-language models such as CLIP, pseudo-labels derived from zero-shot predictions often exhibit significant noise, particularly under domain shifts or in visually complex scenarios. Conventional pseudo-label filtering approaches, which rely on fixed confidence thresholds, tend to be unreliable in fully unsupervised settings. In this work, we propose a novel adaptive pseudo-labeling framework that enhances CLIP's adaptation performance by integrating prototype consistency and neighborhood-based consistency. The proposed method comprises two key components: PICS, which assesses pseudo-label accuracy based on in-class feature compactness and cross-class feature separation; and NALR, which exploits semantic similarities among neighboring samples to refine pseudo-labels dynamically. Additionally, we introduce an adaptive weighting mechanism that adjusts the influence of pseudo-labeled samples during training according to their estimated correctness. Extensive experiments on 11 benchmark datasets demonstrate that our method achieves state-of-the-art performance in unsupervised adaptation scenarios, delivering more accurate pseudo-labels while maintaining computational efficiency.