A Study of Unsupervised Evaluation Metrics for Practical and Automatic Domain Adaptation

TL;DR

This paper introduces the Augmentation Consistency Metric (ACM), an unsupervised evaluation metric for domain adaptation that assesses model quality without target labels, improving hyper-parameter tuning and model selection.

Contribution

The paper proposes a novel unsupervised evaluation metric, ACM, that addresses key issues in existing metrics and enables automatic hyper-parameter optimization in domain adaptation.

Findings

ACM outperforms previous metrics in detecting negative transfer.

Using ACM for hyper-parameter search yields better results than manual tuning.

Large-scale experiments validate the effectiveness of the proposed metric.

Abstract

Unsupervised domain adaptation (UDA) methods facilitate the transfer of models to target domains without labels. However, these methods necessitate a labeled target validation set for hyper-parameter tuning and model selection. In this paper, we aim to find an evaluation metric capable of assessing the quality of a transferred model without access to target validation labels. We begin with the metric based on mutual information of the model prediction. Through empirical analysis, we identify three prevalent issues with this metric: 1) It does not account for the source structure. 2) It can be easily attacked. 3) It fails to detect negative transfer caused by the over-alignment of source and target features. To address the first two issues, we incorporate source accuracy into the metric and employ a new MLP classifier that is held out during training, significantly improving the result. To tackle the final issue, we integrate this enhanced metric with data augmentation, resulting in a novel unsupervised UDA metric called the Augmentation Consistency Metric (ACM). Additionally, we empirically demonstrate the shortcomings of previous experiment settings and conduct large-scale experiments to validate the effectiveness of our proposed metric. Furthermore, we employ our metric to automatically search for the optimal hyper-parameter set, achieving superior performance compared to manually tuned sets across four common benchmarks. Codes will be available soon.