Domain-Specificity Inducing Transformers for Source-Free Domain Adaptation

TL;DR

This paper introduces a novel vision transformer framework that disentangles domain-specific and task-specific features for source-free domain adaptation, achieving state-of-the-art results across multiple benchmarks.

Contribution

It proposes the first use of vision transformers for domain adaptation in a source-free setting, leveraging domain-specific tokens and novel inputs for effective disentanglement.

Findings

Achieves state-of-the-art performance on multiple benchmarks.

Effectively disentangles domain-specific and task-specific factors.

Utilizes domain tokens and domain-representative inputs for adaptation.

Abstract

Conventional Domain Adaptation (DA) methods aim to learn domain-invariant feature representations to improve the target adaptation performance. However, we motivate that domain-specificity is equally important since in-domain trained models hold crucial domain-specific properties that are beneficial for adaptation. Hence, we propose to build a framework that supports disentanglement and learning of domain-specific factors and task-specific factors in a unified model. Motivated by the success of vision transformers in several multi-modal vision problems, we find that queries could be leveraged to extract the domain-specific factors. Hence, we propose a novel Domain-specificity-inducing Transformer (DSiT) framework for disentangling and learning both domain-specific and task-specific factors. To achieve disentanglement, we propose to construct novel Domain-Representative Inputs (DRI) with domain-specific information to train a domain classifier with a novel domain token. We are the first to utilize vision transformers for domain adaptation in a privacy-oriented source-free setting, and our approach achieves state-of-the-art performance on single-source, multi-source, and multi-target benchmarks