Discussion of "Multiple Testing for Exploratory Research" by J. J. Goeman and A. Solari

TL;DR

This paper discusses and extends methods for multiple testing in exploratory research, focusing on providing lower bounds on false hypotheses and identifying false elementary hypotheses, with applications in meta-analysis and flexible exploratory analysis.

Contribution

It analyzes the relation between Goeman and Solari's method and the partial conjunction approach, proposing extensions for more general exploratory settings.

Findings

The method provides a lower bound on false hypotheses.

It can identify false elementary hypotheses when possible.

Extensions allow application in broader exploratory contexts.

Abstract

Goeman and Solari [Statist. Sci. 26 (2011) 584-597, arXiv:1208.2841] have addressed the interesting topic of multiple testing for exploratory research, and provided us with nice suggestions for exploratory analysis. They defined properties that an inferential procedure should have for exploratory analysis: the procedure should be mild, flexible and post hoc. Their inferential procedure gives a lower bound on the number of false hypotheses among the selected hypotheses, and moreover whenever possible identifies elementary hypotheses that are false. The need to estimate a lower bound on the number of false hypotheses arises in various applications, and the partial conjunction approach was developed for this purpose in Biometrics 64 (2008) 1215-1222 (see also Philos. Trans. R. Soc. Lond. Ser. A 367 (2009) 4255-4271 for more details). For example, in a combined analysis of several studies that examine the same problem, it is of interest to give a lower bound on the number of studies in which the finding was reproduced. I will first address the relation between the method of Goeman and Solari and the partial conjunction approach. Then I will discuss possible extensions and address the issue of exploration in more general settings, where the local test may not be defined in advance or where the candidate hypotheses may not be known to begin with.