Support and Invertibility in Domain-Invariant Representations

TL;DR

This paper critically examines the theoretical foundations of domain-invariant representations in unsupervised domain adaptation, highlighting issues with invertibility and strict invariance, and proposes new bounds and practical insights.

Contribution

It introduces generalization bounds that account for non-invertibility and support coverage, challenging existing assumptions and practices in domain-invariant learning.

Findings

Current invariance assumptions can lead to information loss.

Penalizing density differences may be ineffective.

Support coverage is crucial for domain adaptation success.

Abstract

Learning domain-invariant representations has become a popular approach to unsupervised domain adaptation and is often justified by invoking a particular suite of theoretical results. We argue that there are two significant flaws in such arguments. First, the results in question hold only for a fixed representation and do not account for information lost in non-invertible transformations. Second, domain invariance is often a far too strict requirement and does not always lead to consistent estimation, even under strong and favorable assumptions. In this work, we give generalization bounds for unsupervised domain adaptation that hold for any representation function by acknowledging the cost of non-invertibility. In addition, we show that penalizing distance between densities is often wasteful and propose a bound based on measuring the extent to which the support of the source domain covers the target domain. We perform experiments on well-known benchmarks that illustrate the short-comings of current standard practice.