High-Dimensional Kernel Methods under Covariate Shift: Data-Dependent Implicit Regularization

TL;DR

This paper investigates high-dimensional kernel ridge regression under covariate shift, revealing how importance re-weighting acts as a data-dependent regularization to reduce variance and influence bias, with detailed asymptotic analysis.

Contribution

It provides a theoretical framework for understanding the effects of importance re-weighting as a data-dependent regularization in high-dimensional kernel methods under covariate shift.

Findings

Re-weighting reduces variance in kernel ridge regression.

Bias behavior varies with regularization scale.

Spectral decay characterizes bias and variance behavior.

Abstract

This paper studies kernel ridge regression in high dimensions under covariate shifts and analyzes the role of importance re-weighting. We first derive the asymptotic expansion of high dimensional kernels under covariate shifts. By a bias-variance decomposition, we theoretically demonstrate that the re-weighting strategy allows for decreasing the variance. For bias, we analyze the regularization of the arbitrary or well-chosen scale, showing that the bias can behave very differently under different regularization scales. In our analysis, the bias and variance can be characterized by the spectral decay of a data-dependent regularized kernel: the original kernel matrix associated with an additional re-weighting matrix, and thus the re-weighting strategy can be regarded as a data-dependent regularization for better understanding. Besides, our analysis provides asymptotic expansion of kernel functions/vectors under covariate shift, which has its own interest.