Pre-Training Transformers for Domain Adaptation

TL;DR

This paper demonstrates how pre-training transformers like BeiT can effectively transfer knowledge for unsupervised domain adaptation, achieving top results in the Visual Domain Adaptation Challenge 2021.

Contribution

It introduces a semi-supervised approach using BeiT for domain adaptation, outperforming existing state-of-the-art methods.

Findings

Achieved 1st place in the ViSDA Challenge with 56.29% accuracy.

Attained 69.79% AUROC on target datasets.

Showed effectiveness of transformer pre-training for domain transfer.

Abstract

The Visual Domain Adaptation Challenge 2021 called for unsupervised domain adaptation methods that could improve the performance of models by transferring the knowledge obtained from source datasets to out-of-distribution target datasets. In this paper, we utilize BeiT [1] and demonstrate its capability of capturing key attributes from source datasets and apply it to target datasets in a semi-supervised manner. Our method was able to outperform current state-of-the-art (SoTA) techniques and was able to achieve 1st place on the ViSDA Domain Adaptation Challenge with ACC of 56.29% and AUROC of 69.79%.