On the inductive biases of deep domain adaptation

TL;DR

This paper challenges the emphasis on domain invariance in deep domain adaptation, highlighting the importance of hidden inductive biases like pre-training and architecture design, and proposes meta-learning these biases for improved transfer performance.

Contribution

It reveals the limited role of domain invariance and introduces meta-learning of inductive biases to enhance domain adaptation.

Findings

Imposing domain invariance is neither necessary nor sufficient for low target risk.

Hidden inductive biases significantly influence successful domain adaptation.

Meta-learning inductive biases outperforms handcrafted heuristics.

Abstract

Domain alignment is currently the most prevalent solution to unsupervised domain-adaptation tasks and are often being presented as minimizers of some theoretical upper-bounds on risk in the target domain. However, further works revealed severe inadequacies between theory and practice: we consolidate this analysis and confirm that imposing domain invariance on features is neither necessary nor sufficient to obtain low target risk. We instead argue that successful deep domain adaptation rely largely on hidden inductive biases found in the common practice, such as model pre-training or design of encoder architecture. We perform various ablation experiments on popular benchmarks and our own synthetic transfers to illustrate their role in prototypical situations. To conclude our analysis, we propose to meta-learn parametric inductive biases to solve specific transfers and show their superior performance over handcrafted heuristics.