The Role of Embedding Complexity in Domain-invariant Representations

TL;DR

This paper investigates how the complexity of embeddings influences the success of domain-invariant representations in unsupervised domain adaptation, providing theoretical insights and practical strategies for neural networks.

Contribution

It offers a theoretical analysis of embedding complexity's impact on generalization and proposes a method to reduce sensitivity, improving adaptation performance.

Findings

Embedding complexity affects target risk bounds.

Theoretical framework tailored to neural networks.

Empirical results show improved adaptation performance.

Abstract

Unsupervised domain adaptation aims to generalize the hypothesis trained in a source domain to an unlabeled target domain. One popular approach to this problem is to learn domain-invariant embeddings for both domains. In this work, we study, theoretically and empirically, the effect of the embedding complexity on generalization to the target domain. In particular, this complexity affects an upper bound on the target risk; this is reflected in experiments, too. Next, we specify our theoretical framework to multilayer neural networks. As a result, we develop a strategy that mitigates sensitivity to the embedding complexity, and empirically achieves performance on par with or better than the best layer-dependent complexity tradeoff.