The Hitchhiker's Guide to Prior-Shift Adaptation

TL;DR

This paper introduces a new method for adapting classifiers to prior shift in test data, improving accuracy in image classification tasks with changing class distributions.

Contribution

It proposes a novel prior estimation technique addressing issues with confusion matrix-based methods, achieving state-of-the-art results in prior adaptation.

Findings

Proposed method outperforms existing prior estimation techniques.

Achieved 1.1% and 3.4% accuracy improvements in real-world tasks.

Provides best practices for prior shift estimation and classifier adaptation.

Abstract

In many computer vision classification tasks, class priors at test time often differ from priors on the training set. In the case of such prior shift, classifiers must be adapted correspondingly to maintain close to optimal performance. This paper analyzes methods for adaptation of probabilistic classifiers to new priors and for estimating new priors on an unlabeled test set. We propose a novel method to address a known issue of prior estimation methods based on confusion matrices, where inconsistent estimates of decision probabilities and confusion matrices lead to negative values in the estimated priors. Experiments on fine-grained image classification datasets provide insight into the best practice of prior shift estimation and classifier adaptation, and show that the proposed method achieves state-of-the-art results in prior adaptation. Applying the best practice to two tasks with naturally imbalanced priors, learning from web-crawled images and plant species classification, increased the recognition accuracy by 1.1% and 3.4% respectively.