Efficient Hyperparameter Optimization under Multi-Source Covariate Shift

TL;DR

This paper introduces a novel hyperparameter optimization method designed for multi-source covariate shift scenarios, effectively estimating target objectives using unlabeled target data and labeled source data, with theoretical guarantees.

Contribution

It proposes a variance reduced estimator and a hyperparameter optimization procedure tailored for multi-source covariate shift, addressing distributional differences in real-world applications.

Findings

The method achieves no-regret guarantees in hyperparameter tuning.

Experiments show improved performance over naive approaches.

Framework broadens automated hyperparameter optimization applications.

Abstract

A typical assumption in supervised machine learning is that the train (source) and test (target) datasets follow completely the same distribution. This assumption is, however, often violated in uncertain real-world applications, which motivates the study of learning under covariate shift. In this setting, the naive use of adaptive hyperparameter optimization methods such as Bayesian optimization does not work as desired since it does not address the distributional shift among different datasets. In this work, we consider a novel hyperparameter optimization problem under the multi-source covariate shift whose goal is to find the optimal hyperparameters for a target task of interest using only unlabeled data in a target task and labeled data in multiple source tasks. To conduct efficient hyperparameter optimization for the target task, it is essential to estimate the target objective using only the available information. To this end, we construct the variance reduced estimator that unbiasedly approximates the target objective with a desirable variance property. Building on the proposed estimator, we provide a general and tractable hyperparameter optimization procedure, which works preferably in our setting with a no-regret guarantee. The experiments demonstrate that the proposed framework broadens the applications of automated hyperparameter optimization.