# Statistically significant results from low-power analyses: A comedy of errors

## TL;DR

This paper explains how statistically significant results from low-power studies often overestimate the true effect, leading to misleading conclusions.

## Contribution

The paper quantifies estimation bias in low-power analyses and contrasts it with type M error, offering insights into overestimation risks.

## Key findings

- At low power (<30%), significant results strongly overestimate the true effect (relative bias >1.78).
- Sign errors become prevalent only at very low power (<10%).
- Relative bias is less severe than type M error in all scenarios.

## Abstract

When low-power analyses yield statistically significant results, they likely overestimate the true effect. Although sample estimates are symmetrically distributed around the true value, those that are by chance very high are more likely to achieve statistical significance. The bias induced by the significance filter increases as power decreases. Here we sought to quantify the estimation bias associated with low power and to contrast it with the type M error, which assesses the same phenomenon from a different perspective.

We used simulations to quantify estimation bias in relation to power among statistically significant results. We computed the type M error, relative bias (ratio of the estimated mean differences and the true value), and proportions of results with various levels of over- and under-estimation.

For a medium effect size (Cohen's d of 0.5), overestimation of the mean difference was moderate at high power (≥80%): relative bias was <1.13, about 65% of estimates were roughly accurate (between 0.75 and 1.25 of the true value), and sign errors were virtually absent. In contrast, at low power (<30%), overestimation was strong (relative bias >1.78), and almost no estimates were roughly accurate. Sign errors became noticeably prevalent only at very low levels of power (<10%). In all situations, the relative bias had a lower magnitude than the type M error.

Low-power statistically significant results may consist entirely of magnitude errors, sign errors, and type 1 errors with high risk of strong overestimation (double effect). Readers should beware positive results from low-power analyses.

•Among statistically significant results, the true effect is globally overestimated.•This bias increases as the statistical power decreases.•In analyses with a power of 25%, the positive significant effects are twice the true effect.•Readers of research reports should beware positive results of low-power analyses.•Publication decision should be based on the quality of the study, not the results.

Among statistically significant results, the true effect is globally overestimated.

This bias increases as the statistical power decreases.

In analyses with a power of 25%, the positive significant effects are twice the true effect.

Readers of research reports should beware positive results of low-power analyses.

Publication decision should be based on the quality of the study, not the results.