The case for balanced hypothesis tests and equal-tailed confidence intervals

TL;DR

This paper advocates for the use of balanced hypothesis tests and equal-tailed confidence intervals to improve the reliability of directional inference, highlighting issues with current methods and proposing new assessment criteria.

Contribution

It introduces new criteria for evaluating two-sided confidence intervals and hypothesis tests, emphasizing the importance of balanced, equal-tailed approaches for accurate directional interpretation.

Findings

Directional interpretation of some tests can lead to high error rates.

Equal-tailed confidence intervals are preferable to shortest intervals.

Proposed new assessment criteria improve reliability of inference.

Abstract

Introduction: there is an ongoing debate about directional inference of two-sided hypothesis tests for which some authors argue that rejecting $\theta = \theta_0$ does not allow to conclude that $\theta > \theta_0$ or $\theta < \theta_0$ but only that $\theta \neq \theta_0$, while others argue that this is a minor error without practical consequence. Discussion: new elements are brought to the debate. It is shown that the directional interpretation of some non-directional hypothesis tests about Receiver Operating Characteristic (ROC) and survival curves may lead to inflated type III error rates with a probability of concluding that a difference exists in the opposite side of the actual difference that can reach 50% in the worst case. Some of the issues of directional tests also apply to two-sided confidence intervals (CIs). It is shown that equal-tailed CIs should be preferred to shortest CIs. New assessment criteria of two-sided CIs and hypothesis tests are proposed to provide a reliable directional interpretation: partial left-sided and right-sided $\alpha$ error rates for hypothesis tests, probabilities of overestimation and underestimation $\alpha_L$ and $\alpha_U$ and interval half-widths for two-sided CIs. Conclusion: two-sided CIs and two-sided tests are interpreted directionally. This implies that directional interpretation be taken in account in the development and evaluation of confidence intervals and tests.