Prototypical Distillation and Debiased Tuning for Black-box Unsupervised Domain Adaptation

TL;DR

This paper introduces ProDDing, a novel two-step framework for black-box unsupervised domain adaptation that distills knowledge from source models and debiases the target model, outperforming existing methods.

Contribution

ProDDing is the first framework to combine prototypical distillation with debiased tuning specifically for black-box source models in unsupervised domain adaptation.

Findings

ProDDing outperforms existing black-box domain adaptation methods.

ProDDing achieves significant improvements in hard-label black-box scenarios.

Empirical results demonstrate the effectiveness of the proposed approach.

Abstract

Unsupervised domain adaptation aims to transfer knowledge from a related, label-rich source domain to an unlabeled target domain, thereby circumventing the high costs associated with manual annotation. Recently, there has been growing interest in source-free domain adaptation, a paradigm in which only a pre-trained model, rather than the labeled source data, is provided to the target domain. Given the potential risk of source data leakage via model inversion attacks, this paper introduces a novel setting called black-box domain adaptation, where the source model is accessible only through an API that provides the predicted label along with the corresponding confidence value for each query. We develop a two-step framework named $\textbf{Pro}$totypical $\textbf{D}$istillation and $\textbf{D}$ebiased tun$\textbf{ing}$ ($\textbf{ProDDing}$). In the first step, ProDDing leverages both the raw predictions from the source model and prototypes derived from the target domain as teachers to distill a customized target model. In the second step, ProDDing keeps fine-tuning the distilled model by penalizing logits that are biased toward certain classes. Empirical results across multiple benchmarks demonstrate that ProDDing outperforms existing black-box domain adaptation methods. Moreover, in the case of hard-label black-box domain adaptation, where only predicted labels are available, ProDDing achieves significant improvements over these methods. Code will be available at \url{https://github.com/tim-learn/ProDDing/}.