Optimal Inference After Model Selection

TL;DR

This paper introduces a method for valid inference after model selection by controlling the selective type I error, leading to more powerful tests and confidence intervals in exponential family models and linear regression.

Contribution

It develops a framework for post-selection inference that controls selective error rates, deriving most powerful unbiased tests and confidence intervals, including new selective z- and t-tests.

Findings

Derived most powerful unbiased selective tests for exponential families.

Developed new selective z-tests that improve power over existing methods.

Created selective t-tests that do not require error variance knowledge.

Abstract

To perform inference after model selection, we propose controlling the selective type I error; i.e., the error rate of a test given that it was performed. By doing so, we recover long-run frequency properties among selected hypotheses analogous to those that apply in the classical (non-adaptive) context. Our proposal is closely related to data splitting and has a similar intuitive justification, but is more powerful. Exploiting the classical theory of Lehmann and Scheff\'e (1955), we derive most powerful unbiased selective tests and confidence intervals for inference in exponential family models after arbitrary selection procedures. For linear regression, we derive new selective z-tests that generalize recent proposals for inference after model selection and improve on their power, and new selective t-tests that do not require knowledge of the error variance.