# Increased risk of type I errors for detecting heterogeneity of treatment effects in cluster-randomized trials using mixed-effect models

## TL;DR

This paper shows that using standard statistical models in cluster-randomized trials can lead to incorrect conclusions about treatment effect differences across subgroups due to inflated error rates.

## Contribution

The paper identifies a critical issue with type I error inflation in mixed-effect models for HTE analysis in CRTs and proposes a maximal GLMM approach to address it.

## Key findings

- Standard GLMMs can lead to type I error rates as high as 47.2% when within-cluster correlations vary across subgroups.
- Maximal GLMMs and GEE methods better control type I error rates, though GEE still shows slight inflation.
- Model specification significantly impacts inference in real-world CRT data.

## Abstract

Evaluating heterogeneity of treatment effects (HTE) across subgroups is common in both randomized trials and observational studies. Although several statistical challenges of HTE analyses including low statistical power and multiple comparisons are widely acknowledged, issues specific to clustered data, including cluster randomized trials (CRTs), have received less attention. For testing interactions in linear mixed-effects models (LMM), Barr et al. (2013) suggested that: random slopes for interaction terms should be studied. In this paper, we explore the impact of model misspecification, including generalized LMM (GLMM) with or without random slopes, and provide recommendations for conducting inference for HTE across subgroups in CRTs.

We conducted a simulation study to evaluate the performance of common analytic approaches for testing the presence of HTE for continuous, binary, and count outcomes: generalized linear mixed models (GLMM) and generalized estimating equations (GEE) including interaction terms between treatment and subgroup. Several simulation scenarios covered broad range of scenarios in CRTs, for example, small to a large number of clusters, small to moderate cluster-specific random slopes for subgroup. The performance metric was the empirical type I error rate compared to a nominal level. We applied the analytical methods to a real-world CRT using the count outcome utilization of healthcare from the motivating Primary Care Opioid Use Disorder treatment (PROUD) trial.

We found that standard GLMM analyses that assume a common correlation of participants within clusters can lead to severely elevated type 1 error rates of up to 47.2% compared to the 5% nominal level if the within-cluster correlation varies across subgroups. A maximal GLMM, which allows subgroup-specific within-cluster correlations, achieved the nominal type 1 error rate, as did GEE (though rates were slightly elevated even with as many as 50 clusters). Applying the methods to the real-world CRT, we found a large impact of the model specification on inference.

We recommend that HTE analyses using the maximal GLMM account for within-subgroup correlation to avoid anti-conservative inference. For Wald t-testing of HTE in small sample clusters, appropriate small sample correction methods should be considered based on the outcome data type.

The online version contains supplementary material available at 10.1186/s12874-025-02744-6.