Vision Transformer-based Adversarial Domain Adaptation

TL;DR

This paper explores using Vision Transformers as feature extractors in adversarial domain adaptation, demonstrating they can replace CNNs to improve performance in unsupervised domain adaptation tasks.

Contribution

It introduces the use of Vision Transformers in adversarial domain adaptation and shows they can be seamlessly integrated to enhance existing methods.

Findings

ViT can replace CNNs in UDA methods without modification.

Using ViT improves domain adaptation performance.

The approach is plug-and-play and easy to implement.

Abstract

Unsupervised domain adaptation (UDA) aims to transfer knowledge from a labeled source domain to an unlabeled target domain. The most recent UDA methods always resort to adversarial training to yield state-of-the-art results and a dominant number of existing UDA methods employ convolutional neural networks (CNNs) as feature extractors to learn domain invariant features. Vision transformer (ViT) has attracted tremendous attention since its emergence and has been widely used in various computer vision tasks, such as image classification, object detection, and semantic segmentation, yet its potential in adversarial domain adaptation has never been investigated. In this paper, we fill this gap by employing the ViT as the feature extractor in adversarial domain adaptation. Moreover, we empirically demonstrate that ViT can be a plug-and-play component in adversarial domain adaptation, which means directly replacing the CNN-based feature extractor in existing UDA methods with the ViT-based feature extractor can easily obtain performance improvement. The code is available at https://github.com/LluckyYH/VT-ADA.