Label Shift Estimation With Incremental Prior Update
Yunrui Zhang, Gustavo Batista, Salil S. Kanhere

TL;DR
This paper introduces a new post-hoc method for estimating label shift that incrementally updates priors for improved accuracy, outperforming existing maximum likelihood approaches.
Contribution
It proposes a versatile, calibration-weakening approach for label shift estimation that can be applied to any black-box probabilistic classifier, enhancing accuracy.
Findings
Outperforms current state-of-the-art methods on CIFAR-10 and MNIST.
Works under different calibration settings and varying label shift intensities.
Provides a more accurate estimation of label distribution changes over time.
Abstract
An assumption often made in supervised learning is that the training and testing sets have the same label distribution. However, in real-life scenarios, this assumption rarely holds. For example, medical diagnosis result distributions change over time and across locations; fraud detection models must adapt as patterns of fraudulent activity shift; the category distribution of social media posts changes based on trending topics and user demographics. In the task of label shift estimation, the goal is to estimate the changing label distribution in the testing set, assuming the likelihood does not change, implying no concept drift. In this paper, we propose a new approach for post-hoc label shift estimation, unlike previous methods that perform moment matching with confusion matrix estimated from a validation set or maximize the likelihood of the new data with an…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
