Max-Rank: Efficient Multiple Testing for Conformal Prediction

TL;DR

This paper introduces max-rank, a new correction method for multiple hypothesis testing in conformal prediction that exploits dependencies among tests to improve error control and uncertainty quantification.

Contribution

The paper proposes max-rank, a novel correction method that leverages rank order information to enhance multiple testing in conformal prediction, outperforming traditional methods like Bonferroni.

Findings

Max-rank effectively controls family-wise error rate in conformal prediction.

Max-rank demonstrates improved power over Bonferroni correction.

The method is compatible with various conformal procedures.

Abstract

Multiple hypothesis testing (MHT) frequently arises in scientific inquiries, and concurrent testing of multiple hypotheses inflates the risk of Type-I errors or false positives, rendering MHT corrections essential. This paper addresses MHT in the context of conformal prediction, a flexible framework for predictive uncertainty quantification. Some conformal applications give rise to simultaneous testing, and positive dependencies among tests typically exist. We introduce $\texttt{max-rank}$, a novel correction that exploits these dependencies whilst efficiently controlling the family-wise error rate. Inspired by existing permutation-based corrections, $\texttt{max-rank}$ leverages rank order information to improve performance and integrates readily with any conformal procedure. We establish its theoretical and empirical advantages over the common Bonferroni correction and its compatibility with conformal prediction, highlighting the potential to strengthen predictive uncertainty estimates.