A constructive approach to selective risk control

TL;DR

This paper introduces a unified, constructive framework for controlling various selective risks in statistical inference, extending and unifying existing methods like BH and BY procedures, with surprising theoretical insights.

Contribution

It presents a novel iterative approach to control post-selection risks, unifies many existing methods, and extends to multiple risks, with new theoretical results on BH procedure's capabilities.

Findings

BH controls false discovery rate at multiple locations for free

Permutation-based BH can be approximated with near-linear permutations

Many existing methods are special cases of the proposed framework

Abstract

Many modern applications require using data to select the statistical tasks and make valid inference after selection. In this article, we provide a unifying approach to control for a class of selective risks. Our method is motivated by a reformulation of the celebrated Benjamini-Hochberg (BH) procedure for multiple hypothesis testing as the fixed point iteration of the Benjamini-Yekutieli (BY) procedure for constructing post-selection confidence intervals. Building on this observation, we propose a constructive approach to control extra-selection risk (where selection is made after decision) by iterating decision strategies that control the post-selection risk (where decision is made after selection). We show that many previous methods and results are special cases of this general framework, and we further extend this approach to problems with multiple selective risks. Our development leads to two surprising results about the BH procedure: (1) in the context of one-sided location testing, the BH procedure not only controls the false discovery rate at the null but also at other locations for free; (2) in the context of permutation tests, the BH procedure with exact permutation p-values can be well approximated by a procedure which only requires a total number of permutations that is almost linear in the total number of hypotheses.