Statistical Power Analyses for Quantifying the Similarity of Categories of Surgical Procedures Among Pairs of Hospitals and Ambulatory Facilities

TL;DR

This paper shows that only 1% of hospital pairs have very similar surgical procedure distributions, meaning large databases are needed to find enough similar pairs for research.

Contribution

The study introduces a statistical power analysis framework to estimate the number of similar hospital pairs needed for mixed-methods research.

Findings

Only 1% of facility pairs have a similarity index ≥0.80.

A database with 38 organizations has ≥80% chance of finding five similar pairs.

Over 1,000 organizations are needed for an individual hospital to find multiple similar peers.

Abstract

Introduction: Mixed methods are often used to understand organizational associations and differences. For example, one might compare hospitals and ambulatory surgery centers, each described by its relative distribution of cases’ categories of surgical procedures, quantified using anesthesia Current Procedural Terminology (CPT) codes. The similarity of these distributions between facilities can be assessed using a metric akin to a correlation coefficient. Conceptually, identifying similar organizational pairs is feasible, as most U.S. states and Canadian provinces maintain databases containing such administrative data. However, research proposals based on mixed methods may be hindered by the lack of statistical power analysis to determine whether the quantitative phase will yield a sufficient number of similar facilities to support the qualitative phase (i.e., interviews).

Materials and methods: Data were obtained from the American Society of Anesthesiologists' National Anesthesia Clinical Outcomes Registry. The dataset included 12,902,159 cases across 272 procedure categories, performed at 2442 facilities in the United States. The similarity index between facilities ranged from 0 (no overlap in surgical procedures) to 1 (identical distribution of procedures). Values ≥0.80 were considered indicative of high similarity. We estimated the proportion of highly similar facility pairs (similarity index ≥0.80) with low standard errors (<2.0). For each pair, we computed the inverse of the standard normal distribution based on the ratio of the difference from 0.80 to the standard error. The average of these values yielded the mean prevalence of high similarity. This estimated prevalence was then used in power analyses based on the binomial distribution.

Results: Only 1.00% (standard error: 0.01%) of facility pairs had a similarity index ≥0.80. Based on this prevalence, a database would need to include just 38 organizations to have an ≥80% probability of identifying at least five highly similar pairs for interviews. With data from 67 organizations, there would be a ≥95% probability of identifying at least 15 pairs. In contrast, consider an individual organization deciding whether to (a) join a consortium to identify similar organizations for shared strategies, or (b) invest in analysts to explore mandatory state or provincial databases for such purposes. Unless more than 1,000, and ideally more than 2,100, organizations contribute data, the probability of finding multiple highly similar peers may be low.

Conclusions: Investigators can expect a high probability of obtaining sufficient organizational sample sizes for qualitative interviews when using large-scale databases. Although only a small fraction (approximately 1%) of organization pairs exhibit high similarity, the sheer number of potential pairs in state, provincial, and national databases compensates for this. However, for an individual organization seeking to identify peers for qualitative comparison, the chance of finding highly similar matches based on similar surgical procedures is extremely low, unless joining a very large data collective.

Figures