Dealing with multiple testing: To adjust or not to adjust

TL;DR

This paper discusses the logical considerations and debates surrounding whether to apply adjustments for multiple testing in scientific studies, emphasizing conceptual issues over specific methods.

Contribution

It critically examines the reasoning behind multiple testing adjustments, highlighting the importance of logical clarity in their application.

Findings

Highlights the importance of logical reasoning in multiple testing decisions

Questions the automatic application of adjustments without context

Encourages careful consideration of when adjustments are appropriate

Abstract

Multiple testing problems arise naturally in scientific studies because of the need to capture or convey more information with more variables. The literature is enormous, but the emphasis is primarily methodological, providing numerous methods with their mathematical justification and practical implementation. Our aim is to highlight the logical issues involved in the application of multiple testing adjustment.