Marginal Singularity, and the Benefits of Labels in Covariate-Shift

TL;DR

This paper provides a minimax analysis of covariate-shift, showing how the benefit of target labels depends on a transfer-exponent, and introduces an adaptive semi-supervised method that optimally leverages labels based on this exponent.

Contribution

The paper introduces a new minimax framework capturing the benefits of source and target labels under covariate-shift, with a focus on the transfer-exponent , and extends semi-supervised procedures to adapt to it.

Findings

Transfer-exponent  controls the benefit of target labels.

A continuum of regimes from little to significant benefit of target labels.

An adaptive semi-supervised method achieves minimax transfer rates.

Abstract

We present new minimax results that concisely capture the relative benefits of source and target labeled data, under covariate-shift. Namely, we show that the benefits of target labels are controlled by a transfer-exponent $\gamma$ that encodes how singular Q is locally w.r.t. P, and interestingly allows situations where transfer did not seem possible under previous insights. In fact, our new minimax analysis - in terms of $\gamma$ - reveals a continuum of regimes ranging from situations where target labels have little benefit, to regimes where target labels dramatically improve classification. We then show that a recently proposed semi-supervised procedure can be extended to adapt to unknown $\gamma$, and therefore requests labels only when beneficial, while achieving minimax transfer rates.