Algorithm-Dependent Bounds for Representation Learning of Multi-Source Domain Adaptation

TL;DR

This paper introduces information-theoretic bounds for multi-source domain adaptation, analyzing joint distribution alignment and proposing a novel deep algorithm that effectively handles target shift with improved efficiency.

Contribution

It provides the first mutual information-based generalization bounds for supervised and unsupervised multi-source domain adaptation and proposes a new deep algorithm addressing target shift.

Findings

The proposed algorithm achieves competitive performance on target-shifted benchmarks.

Mutual information bounds offer non-vacuous gradient-norm estimates for the algorithm.

The approach improves memory efficiency compared to state-of-the-art methods.

Abstract

We use information-theoretic tools to derive a novel analysis of Multi-source Domain Adaptation (MDA) from the representation learning perspective. Concretely, we study joint distribution alignment for supervised MDA with few target labels and unsupervised MDA with pseudo labels, where the latter is relatively hard and less commonly studied. We further provide algorithm-dependent generalization bounds for these two settings, where the generalization is characterized by the mutual information between the parameters and the data. Then we propose a novel deep MDA algorithm, implicitly addressing the target shift through joint alignment. Finally, the mutual information bounds are extended to this algorithm providing a non-vacuous gradient-norm estimation. The proposed algorithm has comparable performance to the state-of-the-art on target-shifted MDA benchmark with improved memory efficiency.