On the Hardness of Unsupervised Domain Adaptation: Optimal Learners and Information-Theoretic Perspective

TL;DR

This paper investigates the fundamental difficulty of unsupervised domain adaptation by modeling the problem through an information-theoretic lens, introducing a new measure called PTLU to quantify learning hardness.

Contribution

It introduces a novel information-theoretic framework for analyzing UDA hardness, characterizes the optimal learner, and proposes PTLU as a new difficulty measure.

Findings

PTLU serves as a lower bound for the risk of any learner.

PTLU outperforms existing measures in evaluating UDA difficulty.

The framework provides insights into the intrinsic challenges of UDA under covariate shift.

Abstract

This paper studies the hardness of unsupervised domain adaptation (UDA) under covariate shift. We model the uncertainty that the learner faces by a distribution $\pi$ in the ground-truth triples $(p, q, f)$ -- which we call a UDA class -- where $(p, q)$ is the source -- target distribution pair and $f$ is the classifier. We define the performance of a learner as the overall target domain risk, averaged over the randomness of the ground-truth triple. This formulation couples the source distribution, the target distribution and the classifier in the ground truth, and deviates from the classical worst-case analyses, which pessimistically emphasize the impact of hard but rare UDA instances. In this formulation, we precisely characterize the optimal learner. The performance of the optimal learner then allows us to define the learning difficulty for the UDA class and for the observed sample. To quantify this difficulty, we introduce an information-theoretic quantity -- Posterior Target Label Uncertainty (PTLU) -- along with its empirical estimate (EPTLU) from the sample , which capture the uncertainty in the prediction for the target domain. Briefly, PTLU is the entropy of the predicted label in the target domain under the posterior distribution of ground-truth classifier given the observed source and target samples. By proving that such a quantity serves to lower-bound the risk of any learner, we suggest that these quantities can be used as proxies for evaluating the hardness of UDA learning. We provide several examples to demonstrate the advantage of PTLU, relative to the existing measures, in evaluating the difficulty of UDA learning.