The Statistical Cost of Adaptation in Multi-Source Transfer Learning

TL;DR

This paper investigates the fundamental limits of multi-source transfer learning, introducing the intrinsic cost of adaptation to quantify the unavoidable performance gap between bias-agnostic estimators and an oracle with perfect bias knowledge.

Contribution

It characterizes the intrinsic cost of adaptation in multi-source transfer learning, identifies phase transitions, and proposes estimators under structural assumptions to reduce this cost.

Findings

Adaptation is not always possible with multiple sources.

The intrinsic cost of adaptation increases with the number of sources.

Structural assumptions can significantly reduce the adaptation cost.

Abstract

Multi-source transfer learning can improve target-domain estimation by leveraging related source data, but its benefits depend on unknown source-to-target biases. This raises a fundamental question: can a bias-agnostic estimator perform as well as an oracle that knows the true bias configuration? To study this, we introduce the intrinsic cost of adaptation, defined as the smallest worst-case ratio between the risk of any bias-agnostic estimator and the oracle risk. An intrinsic cost of one means oracle performance is achievable without knowing the biases, whereas a larger cost quantifies the unavoidable price of adaptation. Focusing on parametric estimation, we show that multi-source transfer behaves fundamentally differently from the single-source setting: adaptation is not always possible, even with only two sources. For a fixed number of sources, we characterize the intrinsic cost of adaptation and identify a phase transition separating regimes where oracle performance is achievable from those where it is not. As the number of sources grows, we further show that the adaptation cost increases. When adaptation over the full bias configuration space is impossible, additional structure can substantially reduce the cost. We study settings with ordered biases, clustered source parameters, and sufficiently separated non-informative sources, and propose estimators tailored to each regime, with supporting theoretical and empirical results. Overall, our results delineate the statistical limits of multi-source transfer, clarifying when oracle performance is attainable, when structural assumptions help, and when adaptation is fundamentally impossible.