Multiple Testing for Exploratory Research

TL;DR

This paper introduces a novel multiple testing approach for exploratory research that allows users to freely select rejected sets while providing valid confidence statements on false rejections, enabling flexible post hoc analysis.

Contribution

It reverses the traditional roles in multiple testing, focusing on confidence statements for all rejected sets simultaneously, and offers computational shortcuts for practical implementation.

Findings

Provides a framework for flexible post hoc rejection set selection.

Ensures validity of confidence statements for all rejected sets.

Suggests shortcuts to reduce computational complexity.

Abstract

Motivated by the practice of exploratory research, we formulate an approach to multiple testing that reverses the conventional roles of the user and the multiple testing procedure. Traditionally, the user chooses the error criterion, and the procedure the resulting rejected set. Instead, we propose to let the user choose the rejected set freely, and to let the multiple testing procedure return a confidence statement on the number of false rejections incurred. In our approach, such confidence statements are simultaneous for all choices of the rejected set, so that post hoc selection of the rejected set does not compromise their validity. The proposed reversal of roles requires nothing more than a review of the familiar closed testing procedure, but with a focus on the non-consonant rejections that this procedure makes. We suggest several shortcuts to avoid the computational problems associated with closed testing.