Unsupervised optimal deep transfer learning for classification under general conditional shift

TL;DR

This paper introduces a novel transfer learning method under a general conditional shift assumption, leveraging deep neural networks to improve classification accuracy across different data distributions, especially in high-dimensional settings.

Contribution

It proposes a new GCS assumption, a DNN-based estimator for conditional probabilities, and a pseudo-maximum likelihood approach for target label distribution, achieving minimax optimal rates.

Findings

Achieves minimax optimal convergence rates.

Effectively handles high-dimensional data with low-dimensional structure.

Demonstrates superior performance on simulations and Alzheimer's dataset.

Abstract

Classifiers trained solely on labeled source data may yield misleading results when applied to unlabeled target data drawn from a different distribution. Transfer learning can rectify this by transferring knowledge from source to target data, but its effectiveness frequently relies on stringent assumptions, such as label shift. In this paper, we introduce a novel General Conditional Shift (GCS) assumption, which encompasses label shift as a special scenario. Under GCS, we demonstrate that both the target distribution and the shift function are identifiable. To estimate the conditional probabilities ${\bm\eta}_P$ for source data, we propose leveraging deep neural networks (DNNs). Subsequent to transferring the DNN estimator, we estimate the target label distribution ${\bm\pi}_Q$ utilizing a pseudo-maximum likelihood approach. Ultimately, by incorporating these estimates and circumventing the need to estimate the shift function, we construct our proposed Bayes classifier. We establish concentration bounds for our estimators of both ${\bm\eta}_P$ and ${\bm\pi}_Q$ in terms of the intrinsic dimension of ${\bm\eta}_P$ . Notably, our DNN-based classifier achieves the optimal minimax rate, up to a logarithmic factor. A key advantage of our method is its capacity to effectively combat the curse of dimensionality when ${\bm\eta}_P$ exhibits a low-dimensional structure. Numerical simulations, along with an analysis of an Alzheimer's disease dataset, underscore its exceptional performance.