Phase Transition in Nonparametric Minimax Rates for Covariate Shifts on Approximate Manifolds

TL;DR

This paper investigates the fundamental limits of nonparametric regression under covariate shift when data lies near a low-dimensional manifold, revealing a phase transition in estimation rates and proposing adaptive estimators.

Contribution

It establishes new minimax rates for covariate shift with approximate manifolds and introduces adaptive methods that achieve these rates.

Findings

Identifies a phase transition in minimax estimation rates governed by manifold proximity and sample sizes.

Proposes a local polynomial regression estimator that is rate-optimal across the phase transition.

Develops an adaptive procedure that adjusts to unknown smoothness and intrinsic dimension, nearly attaining optimal rates.

Abstract

We study nonparametric regression under covariate shift with structured data, where a small amount of labeled target data is supplemented by a large labeled source dataset. In many real-world settings, the covariates in the target domain lie near a low-dimensional manifold within the support of the source, e.g., personalized handwritten digits (target) within a large, high-dimensional image repository (source). Since density ratios may not exist in these settings, standard transfer learning techniques often fail to leverage such structure. This necessitates the development of methods that exploit both the size of the source dataset and the structured nature of the target. Motivated by this, we establish new minimax rates under covariate shift for estimating a regression function in a general H\"older class, assuming the target distribution lies near -- but not exactly on -- a smooth submanifold of the source. General smoothness helps reduce the curse of dimensionality when the target function is highly regular, while approximate manifolds capture realistic, noisy data. We identify a phase transition in the minimax rate of estimation governed by the distance to the manifold, source and target sample sizes, function smoothness, and intrinsic versus ambient dimensions. We propose a local polynomial regression estimator that achieves optimal rates on either side of the phase transition boundary. Additionally, we construct a fully adaptive procedure that adjusts to unknown smoothness and intrinsic dimension, and attains nearly optimal rates. Our results unify and extend key threads in covariate shift, manifold learning, and adaptive nonparametric inference.