To Aid Scientific Inference, Emphasize Unconditional Compatibility Descriptions of Statistics

TL;DR

This paper advocates for interpreting statistical results in unconditional terms to better account for background assumptions and uncertainties, thereby reducing overconfidence and misleading conclusions in scientific research.

Contribution

It introduces a reinterpretation framework for statistics like P-values that emphasizes unconditional compatibility, integrating assumption uncertainty into primary results without changing calculation methods.

Findings

Unconditional interpretations improve robustness of statistical inferences.

Emphasizing assumption uncertainty prevents overconfident conclusions.

Method enhances statistical training and reporting practices.

Abstract

All scientific interpretations of statistical outputs depend on background (auxiliary) assumptions that are rarely delineated or explicitly interrogated. These include not only the usual modeling assumptions, but also deeper assumptions about the data-generating mechanism that are implicit in conventional statistical interpretations yet are unrealistic in most health, medical and social research. We provide arguments and methods for reinterpreting statistics such as P-values and interval estimates in unconditional terms, which describe compatibility of observations with an entire set of underlying assumptions, rather than with a narrow target hypothesis conditional on the assumptions. Emphasizing unconditional interpretations helps avoid overconfident and misleading inferences in light of uncertainties about the assumptions used to arrive at the statistical results. These include not only mathematical assumptions, but also those about absence of systematic errors, protocol violations, and data corruption. Unconditional descriptions introduce assumption uncertainty directly into the primary statistical interpretations of results, rather than leaving it for the discussion of limitations after presentation of conditional interpretations. The unconditional approach does not entail different methods or calculations, only different interpretation of the usual results. We view use of unconditional description as a vital component of effective statistical training and presentation. By interpreting statistical outputs in unconditional terms, researchers can avoid making overconfident statements based on statistical outputs. Instead, reports should emphasize the compatibility of results with a range of plausible explanations, including assumption violations.