Model-Robust and Adaptive-Optimal Transfer Learning for Tackling Concept Shifts in Nonparametric Regression

TL;DR

This paper introduces a transfer learning method for nonparametric regression that is robust to model misspecification and adaptively achieves optimal convergence rates, especially under concept shifts and limited data.

Contribution

It presents a novel transfer learning procedure with theoretical guarantees that is robust to model misspecification and adaptively attains minimax optimal rates.

Findings

Spectral algorithms with fixed bandwidth Gaussian kernels can achieve minimax rates in misspecified settings.

The proposed method is minimax optimal up to logarithmic factors.

Key factors influencing transfer efficiency are identified and analyzed.

Abstract

When concept shifts and sample scarcity are present in the target domain of interest, nonparametric regression learners often struggle to generalize effectively. The technique of transfer learning remedies these issues by leveraging data or pre-trained models from similar source domains. While existing generalization analyses of kernel-based transfer learning typically rely on correctly specified models, we present a transfer learning procedure that is robust against model misspecification while adaptively attaining optimality. To facilitate our analysis and avoid the risk of saturation found in classical misspecified results, we establish a novel result in the misspecified single-task learning setting, showing that spectral algorithms with fixed bandwidth Gaussian kernels can attain minimax convergence rates given the true function is in a Sobolev space, which may be of independent interest. Building on this, we derive the adaptive convergence rates of the excess risk for specifying Gaussian kernels in a prevalent class of hypothesis transfer learning algorithms. Our results are minimax optimal up to logarithmic factors and elucidate the key determinants of transfer efficiency.