Protected probabilistic classification

TL;DR

This paper introduces methods to safeguard probabilistic classifiers against data distribution shifts, enhancing model robustness in binary classification by leveraging conformal test martingales and expert advice tracking.

Contribution

It presents novel techniques combining conformal test martingales and expert advice tracking to protect probabilistic models from distribution changes.

Findings

Improved robustness of classifiers under distribution shifts

Effective application of conformal test martingales in classification

Enhanced model reliability in real-world scenarios

Abstract

This paper proposes a way of protecting probabilistic prediction models against changes in the data distribution, concentrating on the case of classification and paying particular attention to binary classification. This is important in applications of machine learning, where the quality of a trained prediction algorithm may drop significantly in the process of its exploitation. Our techniques are based on recent work on conformal test martingales and older work on prediction with expert advice, namely tracking the best expert.