Refiltering hypothesis tests to control sign error

TL;DR

This paper proposes a refiltering method for confidence intervals to control the probability of sign errors, especially in low power settings, without assuming dependencies among test statistics.

Contribution

It introduces a second filtering step for confidence intervals that ensures sign error control without relying on dependency assumptions.

Findings

Refiltering confidence intervals reduces sign errors in low power scenarios.

The method does not require assumptions about dependencies among test statistics.

It maintains nominal confidence levels after filtering.

Abstract

A common, though not recommended statistical practice is to report confidence intervals if and only if they exclude a null value of 0. The resulting filtered confidence intervals generally do not have their nominal confidence level. More worryingly, in low power settings their center points will be much farther from zero than the true parameter is and they will frequently lie on the wrong side of zero. Many confidence intervals are constructed using an asymptotically Gaussian parameter estimate accompanied by a weakly consistent estimate of its variance. In these cases, we can subject the given confidence interval(s) to a second filtering step such that the probability of a sign error is controled. This refiltering step retains only those confidence intervals that are sufficiently well separated from the origin. It requires no assumptions on the dependencies among the test statistics.