UDA-Bench: Revisiting Common Assumptions in Unsupervised Domain Adaptation Using a Standardized Framework

TL;DR

This paper introduces UDA-Bench, a standardized framework for fair evaluation of unsupervised domain adaptation methods, revealing key factors affecting their performance and challenging some common assumptions in the field.

Contribution

The paper presents UDA-Bench, a PyTorch framework for standardized evaluation, and provides a comprehensive empirical study on factors influencing UDA effectiveness.

Findings

Benefits of adaptation diminish with advanced backbones

Current methods underutilize unlabeled data

Pre-training data significantly impacts adaptation performance

Abstract

In this work, we take a deeper look into the diverse factors that influence the efficacy of modern unsupervised domain adaptation (UDA) methods using a large-scale, controlled empirical study. To facilitate our analysis, we first develop UDA-Bench, a novel PyTorch framework that standardizes training and evaluation for domain adaptation enabling fair comparisons across several UDA methods. Using UDA-Bench, our comprehensive empirical study into the impact of backbone architectures, unlabeled data quantity, and pre-training datasets reveals that: (i) the benefits of adaptation methods diminish with advanced backbones, (ii) current methods underutilize unlabeled data, and (iii) pre-training data significantly affects downstream adaptation in both supervised and self-supervised settings. In the context of unsupervised adaptation, these observations uncover several novel and surprising properties, while scientifically validating several others that were often considered empirical heuristics or practitioner intuitions in the absence of a standardized training and evaluation framework. The UDA-Bench framework and trained models are publicly available at https://github.com/ViLab-UCSD/UDABench_ECCV2024.