Misplaced Confidence in Observed Power

TL;DR

This paper critiques the misuse of post-hoc power analysis in a clinical trial, emphasizing that such methods can be misleading and do not reliably assess the study's validity.

Contribution

It highlights the fallacy of relying on observed power calculations after data collection to evaluate study results.

Findings

Post-hoc power analyses are often misleading.

Observed power does not reflect true study reliability.

The paper advocates for proper interpretation of statistical results.

Abstract

A recently published randomized controlled trial in JAMA investigated the impact of the selective serotonin reuptake inhibitor, escitalopram, on the risk of major adverse events (MACE). The authors estimated a hazard ratio (HR) of 0.69 (95% CI: 0.49, 0.96; $p$ = 0.03) and then attempted to calculate how much statistical power their study (test) had attained, and used this measure to assess how reliable their results were. Here, we discuss why this approach, along with other post-hoc power analyses, are highly misleading.

Full text

1 Background

Kim et al. [1] present the data from a randomized trial showing that treatment with escitalopram for 24 weeks lowered the risk of major adverse cardiac events (MACE) in depressed patients following recent acute coronary syndrome. The authors should be commended for exploring new approaches to reduce major cardiac events. However, in their paper, they seem to misunderstand the purpose of statistical power. In statistical hypothesis testing, power is the pre-study probability of correctly rejecting the test hypothesis, such as a null hypothesis, given a correct alternative hypothesis [2]. This is a frequentist probability, so it is calculated with multiple replications of the same design in mind. Power analyses can be useful for designing studies but lose their utility after the study data have been obtained and analyzed [3].

In the design phase, Kim et al. [1] used data from the Korea Acute Myocardial Infarction Registry (KAMIR) study [4] and estimated the power of their design to be 70% and 96% for detecting 10% and 15% group differences, respectively, with a sample size of 300 participants. This is often referred to as “design power.” It is based on the study design and beliefs about the true population structure. After completing the study and analyzing the data, the authors attempted to calculate how much power their test had based on the estimated hazard ratio (HR = 0.69, 95% CI: 0.49, 0.96; p = 0.03). They estimated that their test had 89.7% power to detect between-group differences in MACE incidence rates. This is often referred to as “observed power” because it is calculated from the estimated effect size from the study data [3]. This form of power analysis is extremely misleading.

One cannot know the true power of a statistical test because there is no way to be sure that the effect size estimate from the study (HR = 0.69) is the population effect size. Therefore, one cannot be certain that the observed power (89.7%) is the true power of the test. Observed power can often mislead researchers into having a false sense of confidence in their results. Furthermore, observed power, which is a 1:1 function of the P-value, yields no relevant information not already provided by the P-value [3, 5]. This also remains true for studies that are pooled in meta-analysis and then analyzed for average statistical power, which has recently gained momentum. However, this yields unstable interval estimates in even the most ideal of conditions [6].

Statistical power can be useful for designing studies, but once the data from a study have been produced, it is the effect sizes, the P-values, and the confidence/compatibility intervals that contain insightful information. Observed power adds little to this and can be misleading [3, 6, 5]. If one wishes to see how cautious they should be about their point estimate, then they should look at the width of the interval estimates and the parameter values they cover [2, 7].