Conformal testing in a binary model situation

TL;DR

This paper evaluates the performance of conformal testing in detecting distribution changes in binary data generated from Bernoulli distributions, highlighting potential improvements in efficiency over existing methods.

Contribution

It provides a computational assessment of conformal testing in a binary model, identifying areas for efficiency enhancement in change detection.

Findings

Existing conformal test martingales work well in simple cases.

Efficiency of conformal testing can be significantly improved.

Performance evaluation in a model with unknown parameters and changepoint.

Abstract

Conformal testing is a way of testing the IID assumption based on conformal prediction. The topic of this note is computational evaluation of the performance of conformal testing in a model situation in which IID binary observations generated from a Bernoulli distribution are followed by IID binary observations generated from another Bernoulli distribution, with the parameters of the distributions and changepoint unknown. Existing conformal test martingales can be used for this task and work well in simple cases, but their efficiency can be improved greatly.