Unified Conformalized Multiple Testing with Full Data Efficiency

TL;DR

This paper introduces a unified conformal testing framework that maximizes data use for improved power and false discovery rate control, enabling automatic method selection without data splitting.

Contribution

It proposes a comprehensive framework utilizing all data types for conformal testing, enhancing power and providing a systematic design principle for method selection.

Findings

Significantly improves power by better score quality.

Maximizes calibration set size for more accurate p-values.

Demonstrates superior efficiency across diverse scenarios.

Abstract

Conformalized multiple testing offers a model-free way to control predictive uncertainty in decision-making. Existing methods typically use only part of the available data to build score functions tailored to specific settings. We propose a unified framework that puts data utilisation at the centre: it uses all available data-null, alternative, and unlabelled-to construct scores and calibrate p-values through a full permutation strategy. This unified use of all available data significantly improves power by enhancing non-conformity score quality and maximising calibration set size while rigorously controlling the false discovery rate. Crucially, our framework provides a systematic design principle for conformal testing and enables automatic selection of the best conformal procedure among candidates without extra data splitting. Extensive numerical experiments demonstrate that our enhanced methods deliver superior efficiency and adaptability across diverse scenarios.