When few labeled target data suffice: a theory of semi-supervised domain adaptation via fine-tuning from multiple adaptive starts

TL;DR

This paper develops a theoretical framework for semi-supervised domain adaptation (SSDA) using structural causal models, introduces the MASFT algorithm for effective fine-tuning from multiple starts, and demonstrates near-optimal performance with limited labeled target data.

Contribution

It provides a new theoretical analysis of SSDA, proposes the MASFT algorithm for practical adaptation, and extends UDA methods to achieve minimax-optimal results with minimal labeled target data.

Findings

MASFT achieves near-optimal target performance across various distributional shifts.

Extending UDA methods to SSDA under different assumptions is theoretically justified.

Empirical validation confirms the effectiveness of the proposed methods.

Abstract

Semi-supervised domain adaptation (SSDA) aims to achieve high predictive performance in the target domain with limited labeled target data by exploiting abundant source and unlabeled target data. Despite its significance in numerous applications, theory on the effectiveness of SSDA remains largely unexplored, particularly in scenarios involving various types of source-target distributional shifts. In this work, we develop a theoretical framework based on structural causal models (SCMs) which allows us to analyze and quantify the performance of SSDA methods when labeled target data is limited. Within this framework, we introduce three SSDA methods, each having a fine-tuning strategy tailored to a distinct assumption about the source and target relationship. Under each assumption, we demonstrate how extending an unsupervised domain adaptation (UDA) method to SSDA can achieve minimax-optimal target performance with limited target labels. When the relationship between source and target data is only vaguely known -- a common practical concern -- we propose the Multi Adaptive-Start Fine-Tuning (MASFT) algorithm, which fine-tunes UDA models from multiple starting points and selects the best-performing one based on a small hold-out target validation dataset. Combined with model selection guarantees, MASFT achieves near-optimal target predictive performance across a broad range of types of distributional shifts while significantly reducing the need for labeled target data. We empirically validate the effectiveness of our proposed methods through simulations.