Sequential changepoint detection in classification data under label shift

TL;DR

This paper introduces a nonparametric sequential changepoint detection method for label shift in classification data, effectively identifying distribution changes with minimal assumptions and outperforming existing methods in simulations.

Contribution

It proposes a simple, nonparametric detection procedure for label shift that leverages classifier scores and converges to parametric methods as training data increases.

Findings

Outperforms existing detection procedures in simulations.

Leverages classifier training data for estimation.

Converges to parametric detection methods with more training data.

Abstract

Classifier predictions often rely on the assumption that new observations come from the same distribution as training data. When the underlying distribution changes, so does the optimal classification rule, and performance may degrade. We consider the problem of detecting such a change in distribution in sequentially-observed, unlabeled classification data. We focus on label shift changes to the distribution, where the class priors shift but the class conditional distributions remain unchanged. We reduce this problem to the problem of detecting a change in the one-dimensional classifier scores, leading to simple nonparametric sequential changepoint detection procedures. Our procedures leverage classifier training data to estimate the detection statistic, and converge to their parametric counterparts in the size of the training data. In simulations, we show that our method outperforms other detection procedures in this label shift setting.