Optimally tackling covariate shift in RKHS-based nonparametric regression

TL;DR

This paper investigates covariate shift in RKHS-based nonparametric regression, demonstrating that kernel ridge regression with appropriate regularization is minimax optimal under bounded likelihood ratios, and proposing a reweighted approach for unbounded cases.

Contribution

It establishes minimax optimality of KRR under covariate shift with bounded likelihood ratios and introduces a reweighted KRR method for unbounded likelihood ratios, both with theoretical guarantees.

Findings

KRR is minimax rate-optimal under bounded likelihood ratios.

Naive empirical risk minimization is sub-optimal under covariate shift.

Reweighted KRR achieves minimax optimality for unbounded likelihood ratios.

Abstract

We study the covariate shift problem in the context of nonparametric regression over a reproducing kernel Hilbert space (RKHS). We focus on two natural families of covariate shift problems defined using the likelihood ratios between the source and target distributions. When the likelihood ratios are uniformly bounded, we prove that the kernel ridge regression (KRR) estimator with a carefully chosen regularization parameter is minimax rate-optimal (up to a log factor) for a large family of RKHSs with regular kernel eigenvalues. Interestingly, KRR does not require full knowledge of likelihood ratios apart from an upper bound on them. In striking contrast to the standard statistical setting without covariate shift, we also demonstrate that a naive estimator, which minimizes the empirical risk over the function class, is strictly sub-optimal under covariate shift as compared to KRR. We then address the larger class of covariate shift problems where the likelihood ratio is possibly unbounded yet has a finite second moment. Here, we propose a reweighted KRR estimator that weights samples based on a careful truncation of the likelihood ratios. Again, we are able to show that this estimator is minimax rate-optimal, up to logarithmic factors.