Black-box Source-free Domain Adaptation via Two-stage Knowledge Distillation

TL;DR

This paper introduces a two-stage knowledge distillation approach for black-box source-free domain adaptation, enabling effective model adaptation using only source model outputs and target data, thus protecting source data privacy.

Contribution

It proposes a novel two-stage distillation method that improves domain adaptation without access to source data or model internals, addressing privacy concerns.

Findings

Achieves strong performance on cross-domain segmentation tasks.

Effective in scenarios with only black-box source model access.

Simple and flexible framework for privacy-preserving domain adaptation.

Abstract

Source-free domain adaptation aims to adapt deep neural networks using only pre-trained source models and target data. However, accessing the source model still has a potential concern about leaking the source data, which reveals the patient's privacy. In this paper, we study the challenging but practical problem: black-box source-free domain adaptation where only the outputs of the source model and target data are available. We propose a simple but effective two-stage knowledge distillation method. In Stage \uppercase\expandafter{\romannumeral1}, we train the target model from scratch with soft pseudo-labels generated by the source model in a knowledge distillation manner. In Stage \uppercase\expandafter{\romannumeral2}, we initialize another model as the new student model to avoid the error accumulation caused by noisy pseudo-labels. We feed the images with weak augmentation to the teacher model to guide the learning of the student model. Our method is simple and flexible, and achieves surprising results on three cross-domain segmentation tasks.