A Semiparametric Efficient Approach To Label Shift Estimation and Quantification

TL;DR

This paper introduces SELSE, a semiparametric efficient method for estimating label shift in transfer learning, demonstrating superior performance over existing algorithms especially with large test datasets.

Contribution

The paper proposes SELSE, a novel semiparametric efficient estimator for label shift, and proves its optimality among a broad class of quantification algorithms.

Findings

SELSE outperforms existing methods in empirical tests.

SELSE is especially effective with large test sample sizes.

Theoretical proof of SELSE's semiparametric efficiency.

Abstract

Transfer Learning is an area of statistics and machine learning research that seeks answers to the following question: how do we build successful learning algorithms when the data available for training our model is qualitatively different from the data we hope the model will perform well on? In this thesis, we focus on a specific area of Transfer Learning called label shift, also known as quantification. In quantification, the aforementioned discrepancy is isolated to a shift in the distribution of the response variable. In such a setting, accurately inferring the response variable's new distribution is both an important estimation task in its own right and a crucial step for ensuring that the learning algorithm can adapt to the new data. We make two contributions to this field. First, we present a new procedure called SELSE which estimates the shift in the response variable's distribution. Second, we prove that SELSE is semiparametric efficient among a large family of quantification algorithms, i.e., SELSE's normalized error has the smallest possible asymptotic variance matrix compared to any other algorithm in that family. This family includes nearly all existing algorithms, including ACC/PACC quantifiers and maximum likelihood based quantifiers such as EMQ and MLLS. Empirical experiments reveal that SELSE is competitive with, and in many cases outperforms, existing state-of-the-art quantification methods, and that this improvement is especially large when the number of test samples is far greater than the number of train samples.