A Leakage Bound for Confidence Sets after Black-Box Selection

TL;DR

This paper establishes a bound on the noncoverage probability of confidence sets after black-box selection, linking it to the information the selection process reveals about the data.

Contribution

It introduces a novel leakage bound for confidence sets post black-box selection, quantifying the inferential cost in terms of information measures.

Findings

Bound on noncoverage involving total variation distance

Mutual-information bound for black-box selection

Guarantees for noisy screening with Gaussian bounds

Abstract

In many analyses the object reported at the end is not fixed in advance, but is chosen after a preliminary search over variables, subgroups, transformations, models or contrasts. Classical selective-inference methods are most effective when this search can be written as an explicit selection event. This note treats the less structured case in which the selection rule is a black box and inference is required for the target indexed by the selected object. We show that, for any fixed-target confidence procedure, selected-target noncoverage is bounded by the nominal fixed-target noncoverage plus the average total variation distance between the marginal law of the inferential data and its conditional law given the selected object. A mutual-information bound follows immediately. The result recovers sample splitting as the zero-leakage case and gives explicit guarantees for noisy screening through a Gaussian information bound. Thus the inferential cost of black-box selection is quantified by the information that the selected object carries about the inferential sample.