Multiple imputation of multilevel data with single-level models: A fully conditional specification approach using adjusted group means

TL;DR

This paper introduces a new method for handling missing data in multilevel datasets using single-level models and adjusted group means, which performs well in various scenarios.

Contribution

The novel contribution is a fully conditional specification approach using adjusted group means for multilevel data imputation.

Findings

The AGM approach performs well across most scenarios and can outperform conventional multilevel MI in challenging cases.

Simulation studies show the AGM method is reliable in balanced and unbalanced designs with many variables.

The GM approach does not perform as consistently as the AGM approach.

Abstract

Missing data are a common challenge in multilevel designs, and multiple imputation (MI) is often used for handling them. Past research has shown that multilevel MI provides an effective treatment of missing data, so long as the imputation model takes the multilevel structure and the intended analyses into account, and modern methods have been developed that can accommodate even complex types of analyses. However, multilevel MI can be difficult to apply in practice, where the multilevel structure is often not very pronounced or not of immediate interest in the analysis. In these applications, existing methods can become unstable and often struggle to provide reliable results. In this article, we introduce a fully conditional specification (FCS) approach to multilevel MI that combines single-level imputation methods with group means (GM) or adjusted group means (AGM) to accommodate the multilevel structure. Based on theoretical investigations and multiple simulation studies, we evaluated the performance of these methods across balanced and unbalanced designs and with larger numbers of variables. Our findings suggest that the AGM approach – though not the GM approach – performs well across most scenarios we investigated and can even outperform conventional multilevel MI approaches in challenging applications. We also provide an illustrative example of implementing these methods in a simulated setting and discuss the implications of our findings for practice.

Figures

Asymptotic bias in the estimated between- and within-group variance of a single variable in the GM and AGM approach. The true variances were equal to $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau ^2=.20$$\end{document}$ and $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlen

Study 1: Bias of the estimated ICC of *X* and the between-group regression coefficients in the two-level regression of *X* on *Y* ( $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{XY}^{B}$$\end{document}$ ) and *Y* on *X* ( $\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setle

Full text

Missing data are a common challenge in multilevel designs, and multiple imputation (MI; Rubin, 1987) is one of the most commonly recommended methods for handling them (e.g., Schafer and Graham, 2002; see also (Enders, 2022). In MI, researchers aim to “fill in” missing data with plausible replacements generated from the observed data and an imputation model. Past research has shown that MI can be effective at handling missing data in multilevel designs, provided that the underlying imputation model adequately takes the multilevel structure and the intended analysis into account (e.g., Schafer and Yucel, 2001; Yucel, 2008; Grund et al., 2018b; Enders et al., 2016; Lüdtke et al., 2017; Enders et al., 2018; Audigier et al., 2018). This is particularly important for multilevel analyses that involve nonlinear or random effects of predictor variables with missing data, and several methods have since been developed that are specifically tailored to these applications (e.g., Grund et al., 2021; Enders et al., 2020; Erler et al., 2016, 2017; Quartagno and Carpenter, 2022; Goldstein et al., 2014; Keller and Enders, 2023). However, researchers are also often confronted with situations in which the multilevel structure is not very pronounced – or not of immediate interest – and the intended analyses are comparatively simple (e.g., McNeish et al., 2016). In these situations, it can be challenging to apply multilevel MI, because the multilevel models employed therein can be difficult to estimate or computationally unstable (e.g., Andridge, 2011, Audigier et al., 2018, Taljaard et al., 2008; see also van Buuren and Groothuis-Oudshoorn, 2011) and because practical applications often involve a larger number of variables, which have been shown to cause issues in MI more generally (Costantini et al., 2023; Grund et al., 2024; Si et al., 2023).

The present study aims to address this gap by introducing an alternative approach to multilevel MI that accommodates the multilevel structure through single-level models with group means instead of multilevel models. The purpose of this approach is to provide a simple alternative to conventional multilevel MI that remains feasible even in challenging situations. In this context, we focus on applications with relatively simple analyses for multilevel designs, such as multilevel analyses with between- and within-group effects, and single-level analyses for clustered data. The present study aims to provide a theoretical rationale for this approach and evaluate it in multiple simulation studies focused on different scenarios, including balanced and unbalanced designs as well as situations with many variables, where the multilevel structure is often not very pronounced or not of immediate interest (see also McNeish et al., 2016).

In the sections that follow, we first outline the types of multilevel analyses that we consider, followed by an overview of missing data in multilevel designs and multilevel MI. Then, we introduce group-mean-based approaches to multilevel MI and present results from four simulation studies evaluating their performance across different scenarios. Finally, we close with a discussion of our findings and their implications for research practice. All materials related to this article, including the computer code and results from the simulation studies, are available on the OSF platform (https://osf.io/mpfzu/).

Multilevel analysis

Multilevel designs are characterized by a clustered structure in which observations are nested within higher-level observational units, for example, cross-sectional designs with persons nested in groups or longitudinal designs with repeated measurements nested in persons (e.g., Raudenbush and Bryk, 2002, Snijders and Bosker, 2012; see also Singer and Willett, 2003).

Throughout this article, we assume that the multilevel design consists of a two-level structure with persons i ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$i = 1, \ldots , n_j$$\end{document} ) nested in groups j ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$j=1,\ldots ,J$$\end{document} ), and we distinguish between balanced designs with equal-sized groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j = n$$\end{document} ) and unbalanced designs with unequal group sizes. We are primarily interested in simple types of analyses for multilevel data, including multilevel analyses with between- and within-group effects and single-level analyses for clustered data.

Between- and within-group effects

One of the main goals in multilevel analyses is to investigate the between-group and within-group relationships between two or more variables. To simplify the description of these analyses, we focus on the case with two variables X and Y, where Y is considered the outcome, and X is the predictor. In a general two-level model, X and Y can be decomposed as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left[ \begin{array}{c} X_{ij} \\ Y_{ij} \end{array} \right] = \left[ \begin{array}{c} X_{j}^B \\ Y_{j}^B \end{array} \right] + \left[ \begin{array}{c} X_{ij}^W \\ Y_{ij}^W \end{array} \right] \; , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{j}^B$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{j}^B$$\end{document} are the between-group components of X and Y in group j, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{ij}^W$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{ij}^W$$\end{document} denote the within-group deviations of person i in group j (e.g., Rabe-Hesketh et al., 2004; Asparouhov and Muthén, 2006; Lüdtke et al., 2008; Muthén, 1994). The between- and within-group components are typically assumed to follow multivariate normal distributions

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \left[ \begin{array}{c} X_{j}^B \\ Y_{j}^B \end{array} \right] \sim N(\boldsymbol{\mu }, \boldsymbol{\Sigma }_B) \; , \quad \left[ \begin{array}{c} X_{ij}^W \\ Y_{ij}^W \end{array} \right] \sim N(\textbf{0}, \boldsymbol{\Sigma }_W) \; , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\mu }$$\end{document} is a vector of means, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\Sigma }_B = \left[ {\begin{smallmatrix} \tau _X^2 & \tau _\textit{XY} \\ \tau _\textit{XY} & \tau _Y^2 \end{smallmatrix}} \right] $$\end{document} is the between-group covariance matrix, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\Sigma }_{W} = \left[ {\begin{smallmatrix} \sigma _X^2 & \sigma _\textit{XY} \\ \sigma _\textit{XY} & \sigma _Y^2 \end{smallmatrix}} \right] $$\end{document} is the within-group covariance matrix. The between-group components \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{j}^B$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{j}^B$$\end{document} can be understood as latent quantities that represent the differences between each group and the overall means and which correspond to the random intercepts of X and Y in a bivariate empty model (Lüdtke et al., 2008). The intraclass correlations (ICCs) of X and Y are given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \frac{\tau _X^2}{\tau _X^2 + \sigma _X^2}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_Y = \frac{\tau _Y^2}{\tau _Y^2 + \sigma _Y^2}$$\end{document} , respectively, and represent the proportions of variance in X and Y that can be attributed to between-group differences (Snijders and Bosker, 2012).

Two general approaches to multilevel analyses can be distinguished, which we term the latent and the manifest covariate approach (Lüdtke et al., 2008; see also Croon and van Veldhoven, 2007, Shin and Raudenbush, 2010). In the latent covariate approach, a multilevel regression model is specified for Y that includes the latent between- and within-group components of X as predictors:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_{ij} = \beta _0 + \beta _\textit{YX}^W X_{ij}^W + \beta _\textit{YX}^B X_{j}^B + U_{j} + E_{ij} \; , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _\textit{YX}^B$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _\textit{YX}^W$$\end{document} denote the between- and within-group effects of X on Y, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_j$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{ij}$$\end{document} denote normally distributed random intercepts and residuals, respectively. By contrast, in the manifest covariate approach, the model includes the observed (manifest) group means \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{X}_{\bullet j}$$\end{document} and the group-mean-centered values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$(X_{ij} - \bar{X}_{\bullet j})$$\end{document} of X as predictors (Enders and Tofighi, 2007; Hofmann and Gavin, 1998; Kreft et al., 1995):

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_{ij} = \gamma _0 + \gamma _\textit{YX}^W (X_{ij} - \bar{X}_{\bullet j}) + \gamma _\textit{YX}^B \bar{X}_{\bullet j} + U_{j} + E_{ij} \; , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _\textit{YX}^B$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma _\textit{YX}^W$$\end{document} again denote the between- and within-group effects of X on Y but with the between- and within-group components of X now represented by the manifest group means and within-group deviations rather than their latent counterparts.

Despite their conceptual similarity, it is well known that the latent and manifest covariate models differ in terms of the estimated and population values of the between-group regression coefficients (Croon and van Veldhoven, 2007; Lüdtke et al., 2008; Grilli and Rampichini, 2011; Asparouhov and Muthén, 2006). Specifically, in a balanced multilevel design with groups of equal size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j = n$$\end{document} , the population values of the coefficients can be expressed as:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \beta _{YX}^B = \frac{\textit{Cov}(X_j^B,Y_j^B)}{\textit{Var}(X_j^B)} = \frac{\tau _{XY}}{\tau _X^2} \quad , \quad \gamma _{YX}^B = \frac{\textit{Cov}(\bar{X}_{\bullet j},Y_j^B)}{\textit{Var}(\bar{X}_{\bullet j})} = \frac{\tau _{XY} + \sigma _{XY} / n}{\tau _X^2 + \sigma _X^2 / n} \; . \end{aligned}$$\end{document}

The difference between the approaches is because the manifest group means \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{X}_{\bullet j}$$\end{document} represent only an unreliable measure of the latent between-group components, where the reliability of the group means depends on the group size \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} and is given by \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textit{Rel}(\bar{X}_{\bullet j}) = \frac{\tau _X^2}{\tau _X^2 + \sigma _X^2 / n_j}$$\end{document} (Raudenbush and Bryk, 2002). The between-group coefficients obtained with the manifest covariate approach will generally be lower than those in the latent covariate approach, unless the group sizes are large (for additional details, see Lüdtke et al., 2008, Asparouhov and Muthén, 2006, Croon and van Veldhoven, 2007).

Single-level analyses

In addition to multilevel analyses, we also consider single-level analyses for clustered data, in which the relationship between X and Y is investigated without distinguishing between the between- and within-group components, namely

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} Y_{ij} = \beta _0 + \beta _\textit{YX} X_{ij} + E_{ij} \; , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _\textit{YX}$$\end{document} denotes the overall effect of X on Y. In multilevel designs, estimating \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _\textit{YX}$$\end{document} can be useful if the between- and within-group effects are not of immediate interest (McNeish et al., 2016), but statistical inferences about \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _\textit{YX}$$\end{document} generally require corrections to the standard errors to account for the clustered data structure (e.g., cluster-robust standard errors; Cameron and Miller, 2015, Liang and Zeger, 1986).

Naturally, the analyses outlined above could be extended, for example, by including nonlinear and interaction effects of the predictors or by including random slopes of the within-group components of the predictors in multilevel analyses (e.g., Snijders and Bosker, 2012; see also Rockwood, 2020, Shin and Raudenbush, 2024). However, we do not consider these extensions in detail because they have been considered in much of the previous literature on missing data in multilevel designs, and we return to this topic in the Discussion (for additional discussion, see Enders, 2025).

Missing data in multilevel designs

Missing data can occur for a variety of reasons, for example, when participants fail to respond to some items on a questionnaire or drop out of an ongoing study. Three types of missing data mechanisms are often distinguished (Rubin, 1987, 1976): missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). These mechanisms assume that there is a hypothetical complete data set that can be decomposed into an observed and a missing part \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{Y} = (\textbf{Y}_\textit{obs}, \textbf{Y}_\textit{mis})$$\end{document} with an indicator matrix \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{R}$$\end{document} that denotes which elements in \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{Y}$$\end{document} are observed. Under MCAR, the probability of missingness is unrelated to both the observed and the missing data, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P(\textbf{R}|\textbf{Y}) = P(\textbf{R})$$\end{document} . Under MAR, missingness is related to the observed but unrelated to the missing data, once the observed data are taken into account, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P(\textbf{R}|\textbf{Y}) = P(\textbf{R}|\textbf{Y}_\textit{obs})$$\end{document} . Finally, under MNAR, missingness is related to both the observed and the missing data, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P(\textbf{R}|\textbf{Y}) = P(\textbf{R}|\textbf{Y}_\textit{obs}, \textbf{Y}_\textit{mis})$$\end{document} .

In this article, we focus on MI, which requires that the missing data are either MCAR or MAR. Handling missing data under MNAR is still possible but typically requires strong assumptions about the missing data mechanism (e.g., Carpenter et al., 2023).

Multilevel designs can include missing data in many forms, for example, in person- or group-level variables and – for person-level variables – either systematically for all persons in a group or sporadically for only some of the persons (e.g., van Buuren, 2011, Resche-Rigon et al., 2013).

More generally, multilevel designs complicate the treatment of missing data using MI, because the multilevel structure must be accounted for when specifying the imputation model (e.g., Enders et al., 2016). In the following, we provide a short overview of multilevel MI before we focus on group-mean-based approaches to multilevel MI.

Multilevel MI

The main idea of MI is to “fill in” missing values with plausible replacements that are drawn from the posterior predictive distribution of the missing data, given the observed data and an imputation model (Rubin, 1987). This process is repeated multiple times to obtain multiple completed copies of the data; the complete data are then analyzed separately, and the results are pooled to provide a final set of results. An important requirement of MI is that the imputation model must accommodate the data structure and the intended analyses (Meng, 1994). Consequently, in multilevel MI, the imputation model is often specified as a multilevel model that mimics the relevant features of the intended analyses (e.g., between- and within-group effects; Schafer and Yucel, 2002; see also Enders et al., 2016, Lüdtke et al., 2017, Goldstein et al., 2009). In the statistical literature on MI, three strategies for specifying the imputation model can be distinguished (Murray, 2018): joint modeling (JM), sequential modeling (SM), and the fully conditional specification (FCS). In this article, we focus on the FCS approach to MI because it provides the most natural basis for group-mean-based multilevel MI. The main idea of the FCS approach is to treat missing data in multiple variables by specifying a set of univariate imputation models, one for each variable with missing data, using an iterative algorithm that repeatedly steps through this sequence to generate the imputed data (Raghunathan et al., 2001; van Buuren et al., 1999, 2006).

FCS approach to multilevel MI

In the multilevel FCS approach, a separate imputation model is specified for each target variable with missing data, where each model must be specified in such a way that it takes the multilevel structure and intended analyses into account, and a considerable number of studies have provided recommendations for the specification of the imputation models in multilevel FCS (for an overview, see van Buuren, 2018).

For the applications considered in this article, previous research has shown that the imputation models in multilevel FCS should be specified in such a way that the model for each variable includes the other variables as well as their group means as predictors (Enders et al., 2016, 2018; Grund et al., 2018b, 2019; Mistler and Enders, 2017). Such an imputation model can be specified using either latent or manifest means, but we focus on the latter (for additional discussion, see Grund et al., 2018a).

For example, consider the multilevel analyses with between- and within-group effects, and the single-level analysis outlined above and suppose that the two variables both have missing data. Then, an imputation model for each variable could be specified that includes the other variable and its (manifest) group mean as predictors, resulting in the following pair of model equations:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} X_{ij}&\!=\! \beta _{X,0} \!+\! \beta _{X,1} (Y_{ij} \!-\! \bar{Y}_{\bullet j}) + \beta _{X,2} \bar{Y}_{\bullet j} + U_{X,j} + E_{X,ij} \\ Y_{ij}&\!=\! \beta _{Y,0} \!+\! \beta _{Y,1} (X_{ij} \!-\! \bar{X}_{\bullet j}) + \beta _{Y,2} \bar{X}_{\bullet j} + U_{Y,j} + E_{Y,ij} \end{aligned} \; . \end{aligned}$$\end{document}

In this model, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _X$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _Y$$\end{document} denote the regression coefficients of the between- and within-group effects in the equations for X and Y, respectively, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_{X,j}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$U_{Y,j}$$\end{document} denote random intercepts, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{X,ij}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{Y,ij}$$\end{document} denote residuals. As an alternative, the specification could be based on latent instead of manifest group means. However, previous research has indicated that these options – though not mathematically equivalent – tend to yield very similar results, except in extreme cases with very small groups and in which the group sizes are strongly unbalanced (Grund et al., 2018a; Resche-Rigon and White, 2018).

In the FCS approach, the imputation model in Eq. 7 is used to iteratively generate replacement values for each target variable, conditional on the observed and the most recently imputed values of the other variables. For example, for the missing values in X at iteration t, a new set of values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{\textit{mis},ij}^{(t+1)}$$\end{document} would be drawn conditionally on the most recently imputed values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{ij}^{(t)}$$\end{document} , where the group means \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{Y}_{\bullet j}^{(t)}$$\end{document} have been updated to reflect these values (van Buuren et al., 2006; Royston, 2005). Once the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{\textit{mis},ij}$$\end{document} have been replaced with imputed values, the same principle is applied to Y, iterating back and forth between the two variables.

Despite the overall efficacy of multilevel MI, previous research has also shown that the multilevel models involved in multilevel MI can sometimes be difficult to estimate, resulting in convergence problems, biased parameter estimates, and statistical inferences with poor frequentist properties (e.g., Andridge, 2011; Drechsler, 2015; Grund et al., 2018b; Enders et al., 2018). In these situations, alternatives to conventional multilevel MI are needed (see also Drechsler, 2015). These alternatives could be particularly useful in applications with many variables and in which the multilevel structure is not very pronounced (e.g., with small or heterogeneous ICCs).

Group-mean-based approaches to multilevel MI

In the following, we describe two alternative approaches to multilevel MI, which we term the group mean (GM) approach and the adjusted group mean (AGM) approach, respectively. The general idea of these approaches is to simplify the imputation procedure by employing single-level imputation models that use group means rather than random effects to accommodate the multilevel structure.

GM approach

In the GM approach, the specification of the imputation model incorporates the manifest group means of the variables similar to multilevel MI. However, in contrast with multilevel MI, group means are used not only to represent the between-group components of the predictor variables but also for the target variables with missing data. Specifically, with two variables X and Y, the imputation model in the GM approach becomes:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} X_{ij}&\!=\! \beta _{X,0} \!+\! \beta _{X,1} (Y_{ij} \!-\! \bar{Y}_{\bullet j}) \!+\! \beta _{X,2} \bar{Y}_{\bullet j} \!+\! \beta _{X,3} \bar{X}_{\bullet j} \!+\! E_{X,ij}\\ Y_{ij}&\!=\! \beta _{Y,0} \!+\! \beta _{Y,1} (X_{ij} \!-\! \bar{X}_{\bullet j}) \!+\! \beta _{Y,2} \bar{X}_{\bullet j} \!+\! \beta _{Y,3} \bar{Y}_{\bullet j} \!+\! E_{Y,ij} \end{aligned} \; , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{X}_{\bullet j}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{Y}_{\bullet j}$$\end{document} are the group means of X and Y, \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _X$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _Y$$\end{document} denote regression coefficients, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{X,ij}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$E_{Y,ij}$$\end{document} denote residuals as before. Notice that these imputation models are single-level rather than multilevel models, and that the group means of the target variable have been added in place of the random intercepts. These models can again be used to generate imputed values for each variable, with the main addition that these steps now also condition on the most recently imputed values of the target variables themselves. For example, for the missing values in X at iteration t, the imputed values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{\textit{mis},ij}^{(t+1)}$$\end{document} would be drawn conditionally on the most recently imputed values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{ij}^{(t)}$$\end{document} and the group mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{Y}_{\bullet j}^{(t)}$$\end{document} of the predictor as well as the group mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{X}_{\bullet j}^{(t)}$$\end{document} of the target variable, which has been updated to reflect the most recently imputed values.

The main rationale of the GM approach is that the group means of the target variables can be regarded as a proxy for the random intercepts that would be used in multilevel MI (e.g., Raudenbush and Bryk, 2002, p.46). However, this double use of the group means is conceptually problematic because it induces a dependency in the imputation procedure, whereby each person’s imputed value on a given variable depends in part on their previously-imputed value on the same variable (i.e., values are used to “impute themselves”). For this reason, we now introduce an adjustment to this procedure that avoids this dependency, and we return to their theoretical properties shortly.

AGM approach

The AGM approach follows the same idea as the GM approach but instead uses adjusted group means, which are computed individually for each person i by eliminating the person’s own value from the computation (see also Suzuki et al., 2012). Specifically, for a variable Y, the adjusted group mean for person i in group j is computed as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \tilde{Y}_{ij} = \frac{1}{n_j - 1} \sum _{i' \ne \, i} Y_{i'\hspace{-0.2ex}j} \; , \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{i'\hspace{-0.2ex}j}$$\end{document} refers to the values of all persons in group j other than person i. Consequently, the adjusted group means differ from the group means in that they take on different values for each person. Given the adjusted group means, the imputation models in the AGM approach are then specified as follows:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} X_{ij}&\!=\! \beta _{X,0} \!+\! \beta _{X,1} (Y_{ij} \!-\! \bar{Y}_{\bullet j}) \!+\! \beta _{X,2} \bar{Y}_{\bullet j} \!+\! \beta _{X,3} \tilde{X}_{ij} \!+\! E_{X,ij} \\ Y_{ij}&\!=\! \beta _{Y,0} \!+\! \beta _{Y,1} (X_{ij} \!-\! \bar{X}_{\bullet j}) \!+\! \beta _{Y,2} \bar{X}_{\bullet j} \!+\! \beta _{Y,3} \tilde{Y}_{ij} \!+\! E_{Y,ij} \end{aligned} \; , \end{aligned}$$\end{document}

Notice that the imputation model now includes the adjusted group means ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{X}_{ij}$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{Y}_{ij}$$\end{document} ) of the target variables in addition to the group means of the predictors. Apart from this, the procedure for generating the imputed values in the AGM approach is similar to the GM approach. For example, for missing values in X at iteration t, the imputed values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{\textit{mis},ij}^{(t+1)}$$\end{document} would be drawn conditionally on the most recently imputed values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$Y_{ij}^{(t)}$$\end{document} and the group means \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{Y}_{\bullet j}^{(t)}$$\end{document} of the predictor as well as the adjusted group mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{X}_{ij}^{(t)}$$\end{document} of the target variable, which has been updated to reflect the most recently imputed values (excluding the person’s own value). Both the GM and the AGM approaches are implemented in the R package miceadds (Robitzsch et al., 2024).

Rationale and theoretical properties

The GM and AGM approaches aim to simplify the imputation model by replacing multilevel models with single-level models that include (adjusted) group means to accommodate the multilevel structure. The use of group means can be justified by the fact that the group mean \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{Y}_{\bullet j}$$\end{document} is an unbiased – though unreliable – estimator of the true between-group components with reliability equal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textit{Rel}(\bar{Y}_{\bullet j}) = \frac{\tau ^2}{\tau ^2 + \sigma ^2 / n_j}$$\end{document} , which approaches unity as the group size increases (Raudenbush and Bryk, 2002). The same justification holds for the AGM approach, which also avoids the conceptual problems of the GM approach.Fig. 1. Asymptotic bias in the estimated between- and within-group variance of a single variable in the GM and AGM approach. The true variances were equal to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau ^2=.20$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2=.80$$\end{document} . n = group size; MD = proportion of missing data; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau ^2$$\end{document}  = between-group variance; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2$$\end{document}  = within-group variance; GM = single-level MI with group means; AGM = single-level MI with adjusted group means

The statistical properties of the GM and AGM approaches are difficult to investigate analytically due to their inherently iterative nature. To still allow us to do so, we considered a simple scenario with a single variable and a balanced design ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j = n$$\end{document} ), where we focused on the bias in the between-group- ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau ^2$$\end{document} ) and within-group variance ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2$$\end{document} ). In addition, we assumed that the data were MCAR with a fixed proportion of missing data per group (p), where each group included \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_\textit{mis}$$\end{document} missing and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_\textit{obs}$$\end{document} observed values, respectively. As a further simplification, we considered non-iterative versions of the GM and AGM approach, in which the group means were defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\bar{Y}_{\bullet j} = \frac{1}{n_\textit{obs}} \sum _{i=1}^{n_\textit{obs}} Y_{ij}$$\end{document} , and the adjusted group means were defined as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{Y}_{ij} = \frac{1}{n_\textit{obs} - 1} \sum _{i' \! \ne i} Y_{i'\!j}$$\end{document} for cases with observed values and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tilde{Y}_{ij} = \frac{1}{n_\textit{obs}} \sum _{i=1}^{n_\textit{obs}} Y_{ij} = \bar{Y}_{\bullet j}$$\end{document} for the cases with missing values. These versions differ from how these approaches are implemented in software, but they allowed us to derive the bias in closed form. Finally, we assumed that the number of groups was large ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J \rightarrow \infty $$\end{document} ).

The results are summarized in Fig. 1, and the derivation is given in the Appendix. Overall, the results indicated that both the GM and AGM approaches yielded biased estimates of the between- and within-group variances, but the biases differed in size and direction. For the GM approach, the bias was given by

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textit{Bias}(\hat{\tau }^2)= & p \, \frac{n_\textit{obs} + n - 1}{(n-1) n_\textit{obs}} \, \sigma ^2 \quad \text {and} \nonumber \\ \textit{Bias}(\hat{\sigma }^2)= & - p \, \frac{n_\textit{obs} + n - 1}{(n-1) n_\textit{obs}} \, \sigma ^2 \; , \end{aligned}$$\end{document}

which is positive for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau ^2$$\end{document} and negative for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2$$\end{document} . For the AGM approach, the bias was given by

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textit{Bias}(\hat{\tau }^2)= & - p \, \frac{n_\textit{mis} + \frac{n_\textit{mis}-1}{n_\textit{obs}} \gamma + 1}{(n-1)(n_\textit{obs}-1)} \gamma \sigma ^2 \quad \text {and} \nonumber \\ \textit{Bias}(\hat{\sigma }^2)= & p \, \frac{n_\textit{mis} - \gamma + 1}{(n - 1) (n_\textit{obs} - 1)} \gamma \sigma ^2 \; , \end{aligned}$$\end{document}

which is negative for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau ^2$$\end{document} and positive for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2$$\end{document} . The magnitude of the bias depends on the proportion of missing data (p), the group size (n), and the true within-group variance ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2$$\end{document} ). For the AGM approach, the bias further depends on the reliability of the adjusted group means \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\gamma = \frac{\tau ^2}{\tau ^2 + \sigma ^2 / (n_\textit{obs}-1)}$$\end{document} . For both approaches, the bias approaches zero as the group size increases ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n \rightarrow \infty $$\end{document} ). Direct comparisons between the approaches are difficult, but the results indicated that the magnitude of the bias in the AGM approach is typically smaller and approaches zero more quickly with increasing group size as compared with the GM approach (see Fig. 1). However, given the simplified nature of the derivation, the results should only be regarded as a broad indication of their performance.

Extension to unbalanced designs

Although the GM and AGM approaches can be applied to both balanced and unbalanced designs, previous research on multilevel MI has indicated that group-mean-based specifications of the imputation model can lead to biased parameter estimates (e.g., Grund et al., 2018a, Resche-Rigon and White, 2018; see also Carpenter et al., 2023). This is because, in unbalanced designs, the conditional distributions of the variables depend on both the group means and the group sizes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} , and it has been recommended that the group sizes should be included in the imputation model (Grund et al., 2018a). For this reason, we also consider extensions of the GM and AGM approach that explicitly incorporate the group sizes in the imputation model. For the GM approach, the extended model is specified as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \begin{aligned} X_{ij}&= \beta _{X,0} + \beta _{X,1} (Y_{ij} - \bar{Y}_{\bullet j}) + \beta _{X,2} \bar{Y}_{\bullet j} + \beta _{X,3} \bar{X}_{\bullet j}\\&\quad + \beta _{X,4} w_j + \beta _{X,5} w_j \bar{Y}_{\bullet j} + \beta _{X,6} w_j \bar{X}_{\bullet j} + E_{X,ij} \\ Y_{ij}&= \beta _{Y,0} + \beta _{Y,1} (X_{ij} - \bar{X}_{\bullet j}) + \beta _{Y,2} \bar{X}_{\bullet j} + \beta _{Y,3} \bar{Y}_{\bullet j}\\&\quad + \beta _{Y,4} w_j + \beta _{Y,5} w_j \bar{X}_{\bullet j} + \beta _{Y,6} w_j \bar{Y}_{\bullet j} + E_{Y,ij} \end{aligned} \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_j = f(n_j)$$\end{document} represents a function of the group sizes (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_j = 1/n_j$$\end{document} ; see Grund et al., 2018a). The AGM approach can be extended in the same manner. The inclusion of the group sizes and their interaction with the group means effectively allows the between-group relationships between the variables to vary as a function of the group size, which has been shown previously to be effective at reducing1 the bias that can result from group-mean-based approaches to multilevel MI (e.g., Grund et al., 2018a; see also Resche-Rigon and White, 2018).Table 1. Simulated conditions in the simulation studiesDesign factorStudy 1Study 2Study 3aStudy 3bAvg. group size (n)5, 10, 20101010Distribution of group sizes ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} )–uniform (40%, 80%), bimodal (40%, 80%)––No. of groups (J)50, 100, 200, 500, 100050, 200, 100050, 100, 20050Avg. ICC.10, .30.10, .30.10, .30.05Range of ICCs–––0 to .10Between-group correlation ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _B$$\end{document} ).50.50.50.50Within-group correlation ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _W$$\end{document} ).20.20.50.50No. of variables222525, 35, 50No. of variables with MD11allallMD mechanismMCARMCARMCARMCARProportion of MD25%25%25%25%Total no. of conditions302463Note. ICC = intraclass correlation; MD = missing data; MCAR = missing completely at random

In the sections that follow, we consider the performance of the GM and AGM approach in more detail and present the results from four simulation studies focused on different scenarios. Study 1 focused on balanced designs. Study 2 focused on unbalanced designs and whether the approaches and their extensions could accommodate unequal group sizes. Studies 3a and 3b focused on applications with many variables and applications in which the multilevel structure is not very pronounced or not of immediate interest in the analyses.

Study 1

In Study 1, our aim was to investigate the overall performance of the GM and AGM approaches in comparison with multilevel MI in a relatively simple setting but with a relatively broad set of conditions. All files concerning the simulation, including the computer code and results, are provided on the OSF platform. In addition, we also provide an example analysis that illustrates the application of the methods with simulated data in Supplement A of the supplemental online materials and on the OSF platform.

Data generation

Data were generated for two standardized variables X and Y from the bivariate two-level model in Eqs. 1 and 2. The between- and within-group covariance matrices \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\Sigma }_B$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\boldsymbol{\Sigma }_W$$\end{document} were specified by setting the ICCs of X and Y and their between- and within-group correlations ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _B$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _W$$\end{document} ) to specific values. Once the complete data were generated, we induced missing data in X by simulating a latent response variable in accordance with the following linear model

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} R_{ij}^*= \alpha + \lambda Y_{ij} + \delta _{ij} \end{aligned}$$\end{document}

where \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha $$\end{document} is a quantile of the standard normal distribution that corresponds to a given probability of missing data (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\alpha = -0.67$$\end{document} for 25% missing data), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda $$\end{document} is a standardized regression coefficient that determines the missing data mechanism (e.g., \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda = 0$$\end{document} for MCAR data; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\lambda \ne 0$$\end{document} for MAR data), and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\delta _{ij}$$\end{document} is a normally distributed residual with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textit{Var}(\delta _{ij}) = 1 - \lambda ^2$$\end{document} . The values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_{ij}$$\end{document} were set to missing, when \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R_{ij}^*> 0$$\end{document} .

Simulated conditions

An overview of the simulated conditions in Study 1 and the other simulation studies is shown in Table 1. In Study 1, we varied the number of persons per group ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} ), the number of groups (J), and the ICCs of the variables. Specifically, we set the number of persons per group to be equal across groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j = n$$\end{document} ) with small, medium, and large values ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n = 5$$\end{document} , 10, 20) as typically encountered in cross-sectional multilevel research. The number of groups was set to values ranging from very small to very large ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J = 50$$\end{document} , 100, 200, 500, 1000). Finally, the ICCs were set to be equal for X and Y with small ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.10$$\end{document} ) and large values ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.30$$\end{document} ).

All other parameters were set to fixed values. Specifically, we set the between-group correlation to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _B =.50$$\end{document} and the within-group correlation to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho _W =.20$$\end{document} , which corresponds to a scenario with a positive contextual effect of X on Y and vice versa. For the missing data, we generated 25% MCAR data in X. We did not consider MAR data, because we expected that the relative differences between the imputation approaches would be similar for MCAR and MAR data2, and we did not vary the probability of missing data, because previous studies have shown that varying this parameter often does not materially change the results apart from scaling the differences between the different methods. In total, the simulation featured \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3 \times 5 \times 2 = 30$$\end{document} conditions. Each condition was replicated 1000 times.

Imputation

The missing data were imputed with three different imputation approaches: (1) single-level FCS with the GM approach (FCS-GM), (2) single-level FCS with the AGM approach (FCS-AGM), and (3) two-level FCS (FCS-2L). To implement these procedures, we used existing imputation methods from the R packages mice (van Buuren and Groothuis-Oudshoorn, 2011) and miceadds (Robitzsch et al., 2024), and we used passive imputation steps to update the (adjusted) group means in FCS-GM and FCS-AGM. For FCS-GM and FCS-AGM, we used the norm method, and for FCS-2L, we used 2l.continuous. Each procedure was used to generate ten imputations, each after ten iterations of the FCS algorithm (Graham et al., 2007). The number of iterations was determined through visual inspections of convergence plots in five replications of a preliminary run of the simulation. In addition, we also included analyses on the basis of the complete data (CD) and listwise deletion (LD) as additional means of comparison.

Analysis

For the analysis of the data, we considered three sets of analysis models with different parameters of interest. First, we specified an empty model with X as the outcome variable, where the parameter of interest was the ICC of X. Second, we specified a pair of two-level models based on a latent covariate approach Eq. 3 with either X or Y as the outcome and the other variable as the predictor, where the parameters of interest were the regression coefficients for the between- and within-group effects of X on Y and vice versa. Third, we specified a pair of two-level models based on the manifest covariate approach Eq. 4, where we were again interested in the between- and within-group effects. These models were considered because they allowed us to investigate different scenarios, in which (a) the variable affected by missing data was either the predictor or the outcome variable and (b) the between-group components were either represented as manifest (as in the GM and AGM approach) or latent components (as in the data-generating model). All analyses were conducted with the R package lavaan (Rosseel, 2012). The results in MI were pooled using Rubin ’s (1987) rules.Table 2. Study 1: Bias of the estimated ICC of X ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_{X}$$\end{document} )n**JTrueCDLDFCS-GMFCS-AGMFCS-2LSmall ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .10$$\end{document} )5500.100 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.005$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.006$$\end{document} 0.100 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.011}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} 2000.100 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} 0.106 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.008$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} 10000.1000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.109 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 10500.100 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.005$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.006$$\end{document} 0.042 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.008$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} 2000.100 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} 0.045 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} 10000.1000.0000.0000.047 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} 0.00020500.100 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} 0.018 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.005$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} 2000.100 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.021 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 10000.1000.0000.0000.022 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.000Large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .30$$\end{document} )5500.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.009$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.011$$\end{document} 0.070 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.019$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.011$$\end{document} 2000.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.081 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.007$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 10000.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.082 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.007$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 10500.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.012$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.012$$\end{document} 0.023 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.016$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.012$$\end{document} 2000.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} 0.033 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.006$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} 10000.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.0000.036 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} 0.00020500.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} 0.013 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.005$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} 2000.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.016 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 10000.300 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.016 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} Note. Bias values larger than ±10% of the true values are printed in bold. Monte Carlo SEs ranged from 0.000 to 0.002. n = group size; J = number of groups; CD = complete data; LD = listwise deletion; FCS-GM = single-level FCS with group means; FCS-AGM = single-level FCS with adjusted group means; FCS-2L = two-level FCS

Evaluation criteria

For each simulated condition, missing data treatment, and parameter of interest, we evaluated the results in terms of the bias of the parameter estimates and (if applicable) the coverage rates of the 95% confidence intervals. The bias is defined as the average difference between the estimates of a parameter and its true value. The coverage rate represents the proportion of 95% confidence intervals that cover the true value. The true values that were used to compute the bias and coverage rates were derived from the population parameters except for those of the manifest covariate models, where we used the average estimates under CD in place of the true values.3 In addition, we report the Monte Carlo standard error for each set of results (Morris et al., 2019).

Results

Due to the large number of results, we focus on the main findings here. Specifically, we focus on the results for the ICC of X and the regression coefficients of the between-group effect of X on Y in the latent covariate model, and we only summarize the remaining results. The complete set of results can be found in Supplement C of the supplemental materials and on the OSF platform.

Bias

The results for bias of the estimated ICC of X are summarized in Table 2 and Fig. 2 (top row). For CD and LD, the results were essentially unbiased. By contrast, FCS-GM tended to overestimate the ICC of X except in conditions with larger groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n = 20$$\end{document} ) and large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.30$$\end{document} ). FCS-AGM showed a slight tendency for downward bias in conditions with small groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5$$\end{document} ), but this bias was much smaller than in FCS-GM and reduced to near zero as the number of groups increased. Finally, FCS-2L provided essentially unbiased results.

For the estimated between-group effects in the latent covariate models, the results for bias are summarized in Table 3 and Fig. 2 (middle and bottom row). The results followed a different pattern depending on whether X was the outcome or the predictor. When X was the predictor ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{YX}^B$$\end{document} ), CD and LD showed a slight upward bias in conditions with small groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n \le 10$$\end{document} ) and small ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.10$$\end{document} ), but this bias reduced to near zero as the number of groups increased. FCS-GM generally tended to underestimate this between-group effect except in conditions with large groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n = 20$$\end{document} ) and large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.30$$\end{document} ). Finally, FCS-AGM and FCS-2L tended to show a slight upward bias in conditions with small groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n \le 10$$\end{document} ), similar to CD and LD, which again reduced to negligible proportions as the number of groups increased. By contrast, when X was the outcome ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{XY}^B$$\end{document} ), all methods performed similarly and showed a slight upward bias in the between-group effects that reduced to near zero as the number of groups increased.

For the other parameters of interest, the results can be summarized as follows (for the full results, see Supplement C). For the estimated within-group effects in the latent covariate models, we generally observed very little bias, except for FCS-GM, which tended to slightly overestimate the within-group effect of X on Y in conditions with small groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5$$\end{document} ). For the manifest covariate models, the results followed essentially the same pattern as in the latent covariate models, except that there was no small-sample bias in the between-group effects for all methods except FCS-GM, which still exhibited the same pattern of bias.Fig. 2. Study 1: Bias of the estimated ICC of X and the between-group regression coefficients in the two-level regression of X on Y ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{XY}^{B}$$\end{document} ) and Y on X ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{XY}^{B}$$\end{document} , latent covariate). n = group size; J = number of groups; \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X, \textrm{ICC}_Y$$\end{document}  = ICCs of X and Y; CD = complete data; LD = listwise deletion; FCS-GM = single-level FCS with group means; FCS-AGM = single-level FCS with adjusted group means; FCS-2L = two-level FCSTable 3Study 1: Bias of the estimated between-group regression coefficient ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{YX}^{B}$$\end{document} ) in the two-level regression of Y on X (latent covariate)n**JTrueCDLDFCS-GMFCS-AGMFCS-2LSmall ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .10$$\end{document} )5500.5000.069****0.097 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.272}$$\end{document} 0.254****0.1432000.5000.0240.053 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.265}$$\end{document} 0.108****0.07610000.5000.0020.006 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.267}$$\end{document} 0.0280.00910500.5000.059****0.093 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.144}$$\end{document} 0.167****0.1102000.5000.0080.012 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.160}$$\end{document} 0.0310.01410000.5000.0060.007 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.158}$$\end{document} 0.0210.00820500.5000.0000.005 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.096}$$\end{document} 0.0220.0072000.5000.0030.002 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.091}$$\end{document} 0.0110.00310000.5000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.091}$$\end{document} 0.0070.000Large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .30$$\end{document} )5500.5000.0230.037 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.099}$$\end{document} 0.0630.0442000.5000.0010.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.115}$$\end{document} 0.0130.00210000.5000.0020.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.114}$$\end{document} 0.0110.00110500.5000.0100.009 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.052}$$\end{document} 0.0180.0092000.5000.0020.004 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.054}$$\end{document} 0.0080.00310000.5000.0000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf {-0.056}$$\end{document} 0.0050.00020500.5000.0070.009 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.020$$\end{document} 0.0120.0082000.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.028$$\end{document} 0.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 10000.5000.0010.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.026$$\end{document} 0.0040.002Note. Bias values larger than ±10% of the true values are printed in bold. Monte Carlo SEs ranged from 0.001 to 0.043. n = group size; J = number of groups; CD = complete data; LD = listwise deletion; FCS-GM = single-level FCS with group means; FCS-AGM = single-level FCS with adjusted group means; FCS-2L = two-level FCSTable 4Study 1: Coverage rates (in %) of the 95% confidence intervals of the estimated between-group regression coefficient ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{YX}^{B}$$\end{document} ) in the two-level regression of Y on X (latent covariate)n**JTrueCDLDFCS-GMFCS-AGMFCS-2LSmall ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .10$$\end{document} )5500.50094.394.181.096.195.72000.50093.393.238.094.094.310000.50094.495.10.194.693.310500.50091.592.389.995.194.42000.50095.094.465.895.294.610000.50096.494.55.594.694.820500.50094.695.492.994.995.02000.50094.795.183.194.294.810000.50093.593.635.793.993.8Large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .30$$\end{document} )5500.50094.093.590.193.893.82000.50095.893.873.695.594.510000.50094.195.114.894.494.610500.50093.094.793.594.694.42000.50093.594.788.894.194.410000.50094.994.162.093.694.520500.50094.693.694.393.693.92000.50094.694.292.994.194.010000.50096.295.288.695.895.9Note. Coverage rates below 92.5% are printed in bold. Monte Carlo SEs ranged from 0.1% to 1.5%. n = group size; J = number of groups; CD = complete data; LD = listwise deletion; FCS-GM = single-level FCS with group means; FCS-AGM = single-level FCS with adjusted group means; FCS-2L = two-level FCS

Coverage rates

For the coverage rates, we focus on the results for the between-group effects of X on Y in the latent covariate model, which are summarized in Table 4. Overall, the results followed a pattern similar to the bias. Specifically, for CD and LD, we observed coverage rates close to the nominal value of 95%, except in one condition with medium-sized groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=10$$\end{document} ), small ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.10$$\end{document} ), and a small number of groups. For FCS-GM, the coverage rates were often noticeably below the nominal value, especially in conditions with small groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n \le 10$$\end{document} ) and small ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.10$$\end{document} ). FCS-AGM and FCS-2L showed near-nominal coverage rates in all conditions.

The results for the coverage rates of the other parameters also closely tracked the results for the bias and can be summarized as follows (see Supplement C). For the between-group effects of Y on X in the latent covariate model, the coverage rates for all methods were close to nominal. For the within-group effects, the coverage rates were close to nominal for all methods except FCS-GM, which showed coverage rates slightly below the nominal value in conditions with small groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=5$$\end{document} ). Finally, for the manifest covariate models, the results followed the same pattern, except for LD, which sometimes showed below-nominal coverage rates for the between-group effects.

Summary

The results of Study 1 indicated that the AGM approach (FCS-AGM) can provide accurate results that are similar to those obtained with multilevel MI (FCS-2L) in applications with balanced designs. By contrast, the GM approach (FCS-GM) provided more biased results, and this bias often persisted in large samples with many groups. These results were also consistent with the theoretical properties of the GM and AGM approaches. Because our focus in Study 1 was on relatively simple applications with balanced data, our focus in Study 2 was on applications in unbalanced designs with unequal group sizes.

Study 2

In Study 2, we focused on unbalanced designs and aimed to evaluate the performance of the GM and AGM approaches in applications in which the group sizes were imbalanced in different ways and to different degrees. To this end, we expanded the simulation design to include unequally-sized groups, and we included the extended versions of the GM and AGM approaches for unbalanced designs.

Data generation

To generate the data, we adapted the procedure from Study 1 to feature unbalanced designs with unequal group sizes. Specifically, given an average group size n, we generated the group sizes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} in such a way that their distribution reflected one of two cases: a uniform case or a binary case. In the uniform case, we simulated group sizes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} that were evenly distributed in a symmetric interval around n (e.g., with an 80% interval around \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=10$$\end{document} , which would result in group sizes from 2 to 18). In the binary case, we simulated group sizes \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} that were split evenly between only the extreme values of this interval (e.g., with an 80% interval around \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=10$$\end{document} , which would result in half of the groups having sizes of 2 and 18, respectively).

The simulated conditions that we varied included the number of groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=50$$\end{document} , 200, 1000), the ICCs of the variables ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.10$$\end{document} , .30), and the size of the interval of the group sizes around the average group size (for an overview, see Table 1).

Specifically, we set the average group size to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=10$$\end{document} and simulated group sizes in 40% and 80% intervals around this value, which we refer to as U40 and U80 in the uniform case and B40 and B80 in the binary case. The other population parameters were fixed to the same values as in Study 1. In total, the simulation featured \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3 \times 2 \times 4 = 24$$\end{document} conditions, each replicated 1000 times.

Imputation and analysis

To impute the missing data, we used the same methods as in Study 1 and also included the extensions of the GM and AGM approach for unbalanced designs (henceforth called FCS-GM* and FCS-AGM*). In the extended versions (FCS-GM* and FCS-AGM*), the group sizes were incorporated in the imputation model as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_j = 1 / n_j$$\end{document} (see Eq. 13). We again used the mice and miceadds packages to implement all procedures. The analysis models were the same as in Study 1 and included the empty model with X as the outcome as well as the latent and manifest covariate models with X as the predictor and Y as the outcome and vice versa. The models were again fitted with the lavaan package, and we used the same evaluation criteria as before.Table 5. Study 2: Bias of the estimated between-group regression coefficients ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{YX}^{B}$$\end{document} ) in the two-level regression of Y on X (latent covariate) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} JTrueCDLDFCS-GMFCS-GMFCS-AGMFCS-AGMFCS-2LSmall ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .10$$\end{document} )U40500.5000.057****0.084 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.141}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.143}$$\end{document} 0.1760.1520.1192000.5000.0080.012 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.161}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.161}$$\end{document} 0.0350.0300.01510000.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 0.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.163}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.163}$$\end{document} 0.0160.0140.001B40500.5000.0190.037 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.157}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.158}$$\end{document} 0.1100.0900.0622000.5000.0090.014 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.160}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.160}$$\end{document} 0.0380.0270.01510000.5000.0040.004 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.163}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.162}$$\end{document} 0.0230.0160.004U80500.5000.0250.041 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.155}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.165}$$\end{document} 0.1220.1230.0672000.5000.0060.013 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.160}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.161}$$\end{document} 0.0480.0310.01510000.5000.0010.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.163}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.163}$$\end{document} 0.0290.0130.000B80500.5000.0210.044 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.151}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.156}$$\end{document} 0.1590.0840.0562000.5000.0100.013 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.151}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.151}$$\end{document} 0.0900.0290.01510000.5000.0010.002 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.155}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.154}$$\end{document} 0.0700.010 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} Large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .30$$\end{document} )U40500.5000.0150.018 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.044$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.046$$\end{document} 0.0260.0250.0192000.5000.0020.002 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.057}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.057}$$\end{document} 0.0070.0080.00310000.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.059}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.059}$$\end{document} 0.0020.003 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} B40500.5000.0120.010 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.054}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.054}$$\end{document} 0.0190.0190.0122000.5000.0060.006 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.058}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.058}$$\end{document} 0.0080.0120.00610000.5000.0010.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.062}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.062}$$\end{document} 0.0020.0050.001U80500.5000.0000.005 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.066}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.070}$$\end{document} 0.0120.0060.0052000.5000.0050.006 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.061}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.060}$$\end{document} 0.0110.0130.00810000.5000.0000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.067}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.066}$$\end{document} 0.0030.0050.001B80500.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} 0.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.099}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.098}$$\end{document} 0.0160.0060.0002000.5000.0010.002 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.096}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.090}$$\end{document} 0.0120.0080.00410000.5000.0010.002 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.096}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.088}$$\end{document} 0.0100.0090.004Note. Bias values larger than ±10% of the true values are printed in bold. Monte Carlo SEs ranged from 0.001 to 0.024. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} = group size distribution; U40/80 = uniform in a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40/80\%$$\end{document} interval around n; B40/80 = bimodal with values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40/80\%$$\end{document} above and below n; J = number of groups; CD = complete data; LD = listwise deletion; FCS-GM() = single-level FCS with group means (and group sizes); FCS-AGM() = single-level FCS with adjusted group means (and group sizes); FCS-2L = two-level FCS

Results

Due to the large number of results, we focus on the results for the between-group effects in the latent covariate model and only summarize the remaining findings (for the full results, see Supplement C). Regarding the bias, the results for the between-group effects of X on Y in the latent covariate model are shown in Table 5. For CD and LD, we sometimes observed a slight upward bias, but this finding was not consistent across the types of imbalance and limited to small samples ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=50$$\end{document} ). For FCS-GM, we again observed biased estimates, especially in conditions with small ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y =.10$$\end{document} ). This was also true for FCS-GM*, which did not noticeably reduce bias. For FCS-AGM, we again found bias in the opposite direction of FCS-GM, but this bias again tended to be smaller and decrease further as the number of groups increased, except in the most extreme imbalance condition (B80). FCS-AGM* showed a further reduction in bias and reduced the bias to negligible proportions as the number of groups increased. Finally, for FCS-2L, the bias followed a pattern similar to FCS-AGM*. For the other parameters of interest, the results followed the same pattern as in Study 1 and were also affected by the unbalanced design in the same way (see Supplement C).Table 6. Study 2: Coverage rates (in %) of the 95% confidence intervals of the estimated between-group regression coefficients ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{YX}^{B}$$\end{document} ) in the two-level regression of Y on X (latent covariate) \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} JTrueCDLDFCS-GMFCS-GMFCS-AGMFCS-AGMFCS-2LSmall ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .10$$\end{document} )U40500.50093.393.090.5****90.795.195.094.92000.50093.993.068.4****68.194.294.693.810000.50095.495.74.2****3.794.995.795.5B40500.50092.291.687.5****86.893.694.493.22000.50094.093.667.5****68.793.893.893.710000.50095.294.66.3****6.793.693.493.7U80500.50092.893.189.6****89.394.894.193.72000.50094.893.967.6****67.393.694.193.910000.50095.094.47.4****7.193.094.294.8B80500.50091.892.088.2****87.794.594.294.22000.50094.593.874.6****76.192.694.794.510000.50095.294.812.914.183.693.894.4Large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_X = \textrm{ICC}_Y = .30$$\end{document} )U40500.50092.892.592.792.392.593.292.62000.50093.493.587.3****86.992.693.593.210000.50096.195.660.5****60.495.896.096.1B40500.50093.093.492.893.493.794.194.12000.50095.194.988.8****88.894.894.694.810000.50095.695.356.4****57.695.695.095.6U80500.50094.493.593.292.994.993.893.82000.50094.894.989.7****89.893.694.294.710000.50094.293.653.8****55.394.094.493.7B80500.50094.193.791.1****91.492.793.893.62000.50095.294.681.5****83.293.293.994.210000.50093.294.031.7****40.293.293.993.6Note. Coverage rates below 92.5% are printed in bold. Monte Carlo SEs ranged from 0.6% to 1.6%. \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n_j$$\end{document} = group size distribution; U40/80 = uniform in a \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40/80\%$$\end{document} interval around n; B40/80 = bimodal with values \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$40/80\%$$\end{document} above and below n; J = number of groups; CD = complete data; LD = listwise deletion; FCS-GM() = single-level FCS with group means (and group sizes); FCS-AGM() = single-level FCS with adjusted group means (and group sizes); FCS-2L = two-level FCS

Regarding the coverage rates, the results for the between-group effects of X on Y in the latent covariate model are shown in Table 6. The overall pattern of results was again similar to the bias. Specifically, for CD and LD, the coverage rates were typically close to the nominal value of 95%. For FCS-GM and FCS-GM*, we often observed coverage rates that were noticeably below the nominal value. For FCS-AGM, the coverage rates were close to the nominal value, except in one condition with extremely imbalanced groups (B80). Finally, FCS-AGM* and FCS-2L showed near-nominal coverage rates in all conditions. The results for the coverage rates for the other parameters followed the same pattern as in Study 1 and showed the same impact of the unbalanced design as the ones for the bias (see Supplement C).

Summary

The results of Study 2 indicated that the AGM approach (FCS-AGM) can also provide results that are similar to multilevel MI (FCS-2L) in unbalanced designs. This was often true even for the unmodified version of the AGM approach, in which the imputation model did not include the group sizes, so long as the degree of imbalance was not too extreme. For extremely imbalanced group sizes, the performance of the unmodified AGM approach became worse, but the extended version of the AGM approach (FCS-AGM*) continued to provide results that were similar to multilevel MI. However, all of the results presented so far were based on scenarios with a small number of variables and a multilevel structure that is at least somewhat pronounced, in which both group-mean-based and conventional approaches to multilevel MI should be expected to perform relatively well. For this reason, in Studies 3a and 3b, we focused on more challenging scenarios with small samples, many variables, and less pronounced multilevel structures.

Studies 3a and 3b

Both Studies 3a and 3b were concerned with more challenging scenarios but placed different emphases in terms of sample size, the number of variables, and the strength of the multilevel structure. In Study 3a, we considered scenarios with a larger number of variables, in which the aim was to conduct both multilevel and single-level analyses. In Study 3b, we considered more extreme scenarios with an even larger number of variables and a multilevel structure that was weaker and more heterogeneous.

Data generation

The data-generating procedure was similar to Study 1 but was adapted to feature a larger number of variables with a different pattern of between-group and within-group relationships. Specifically, we simulated data for multiple standardized variables \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textbf{X} = (X_1, \ldots , X_{K})$$\end{document} using a multivariate two-level model:

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \textbf{X}_{j}^B \sim N(\textbf{0}, \boldsymbol{\Sigma }_B) \; , \quad \textbf{X}_{ij}^W \sim N(\textbf{0}, \boldsymbol{\Sigma }_W) \; . \end{aligned}$$\end{document}

For simplicity, we assumed that the correlations between the variables were the same for all variables, and we set the between- and within-group correlations to be equal so that the multilevel structure no longer featured contextual effects. Specifically, the between- and within-group covariance matrices were defined as

\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\begin{aligned} \boldsymbol{\Sigma }_B= & \left[ \begin{array}{cccc} \tau _1^2 & \rho \tau _{1} \tau _{2} & \cdots & \rho \tau _1\tau _K \\ \rho \tau _1\tau _2 & \tau _2^2 & & \rho \tau _2\tau _K \\ \vdots & & \ddots & \vdots \\ \rho \tau _1\tau _K & \rho \tau _2\tau _K & \cdots & \tau _K^2 \end{array} \right] \; \text {and} \nonumber \\ \boldsymbol{\Sigma }_W= & \left[ \begin{array}{cccc} \sigma _1^2 & \rho \sigma _{1} \sigma _{2} & \cdots & \rho \sigma _1\sigma _K \\ \rho \sigma _1\sigma _2 & \sigma _2^2 & & \rho \sigma _2\sigma _K \\ \vdots & & \ddots & \vdots \\ \rho \sigma _1\sigma _K & \rho \sigma _2\sigma _K & \cdots & \sigma _K^2 \end{array} \right] \; , \end{aligned}$$\end{document}

where the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau ^2$$\end{document} and the \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\sigma ^2$$\end{document} are the between- and within-group variances of the variables. Once the data were generated, we simulated the missing data as before. However, in contrast to the previous studies, we generated missing data on all variables, where each variable had the same probability of missing data.

Simulated conditions

The simulated conditions in Studies 3a and 3b had different emphases and are summarized in Table 1. In Study 3a, we fixed the group sizes to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$n=10$$\end{document} , the number of variables to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=25$$\end{document} , the between- and within-group correlation to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\rho =.50$$\end{document} , and the proportion of missing data per variable to 25% in accordance with an MCAR mechanism. The factors we varied were the number of groups (J = 50, 100, 200) and the ICCs, whose values we set to be homogeneous across variables ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_{X_1} = \ldots = \textrm{ICC}_{X_{25}}$$\end{document}  = .10, .30). These values were chosen to mimic applications in which the main challenge is that the number of variables is relatively large in comparison with the sample size. In Study 3b, we then varied the number of variables from moderate to very large (K = 25, 35, 50) and specified fixed values for the ICCs, which we set to be heterogeneous in a range from 0 to .10. To this end, we divided the variables into five groups of five (for \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=25$$\end{document} ), seven ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=35$$\end{document} ), or ten ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=50$$\end{document} ) variables, respectively, and set the ICCs of the variables in each group to .10, .075, .05, .025, and 0, respectively. The number of groups was set to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=50$$\end{document} , and the other factors were set to the same values as in Study 3a. These values were chosen because they reflect the conditions encountered in practice, where the number of variables is often relatively large, and the ICCs of the variables are often heterogeneous e.g.,, (Hedges and Hedberg, 2007; Brunner et al., 2018). Study 3a thus featured \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$3 \times 2 = 6$$\end{document} conditions, and Study 3b featured three conditions. Each condition was replicated 1000 times.

Imputation and analysis

In Studies 3a and 3b, we used the same imputation procedures as in Study 1 (FCS-GM, FCS-AGM, and FCS-2L). To implement them, we again used the mice and miceadds packages, but we increased the number of iterations to 20. We also included CD but did not include LD due to the large proportion of cases with missing data. For the analysis of the data, we then considered five analysis models. First, we specified an empty model with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} as the outcome variable, where the parameter of interest was the ICC of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_{X_1}$$\end{document} ). The second and third models were the latent and manifest covariate models with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} as the outcome and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} as the predictor. The fourth model was a single-level regression model with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} as the outcome and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} as the predictor. Finally, the fifth model was a single-level regression model with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} as the outcome and all other variables ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2, \ldots , X_{K}$$\end{document} ) as predictors. All analyses were conducted using lavaan. For the single-level models, we used cluster-robust standard errors to account for the multilevel structure (Cameron and Miller, 2015).Table 7. Study 3a: Bias and coverage rates (in %) of the 95% confidence intervals of the estimated regression coefficients ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} ) in the single- level regressions of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2, \ldots , X_{25}$$\end{document} (cluster-robust SEs) and the estimated between-group coefficient ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}^{B}$$\end{document} ) in the two-level regression of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} (latent covariate)BiasCoverage (%)JTrueCDFCS-GMFCS-AGMFCS-2LCDFCS-GMFCS-AGMFCS-2LSmall ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_{X_1} = \ldots = \textrm{ICC}_{X_{25}} = .10$$\end{document} )Single-level regression: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} )500.5000.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.016$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.016$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.047$$\end{document} 94.694.994.497.12000.5000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.005$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.004$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.007$$\end{document} 94.695.094.496.810000.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.002$$\end{document} 95.794.594.497.2Single-level regression: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2, \ldots , X_{25}$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} )500.0400.0020.0030.0040.00392.894.095.698.22000.0400.0000.0000.0000.00093.894.094.194.410000.040 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 94.493.894.294.5Two-level regression (latent covariate): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}^B$$\end{document} )500.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.125}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.036$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.132}$$\end{document} 95.396.197.599.32000.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.109}$$\end{document} 0.007 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.018$$\end{document} 95.489.695.598.310000.5000.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.105}$$\end{document} 0.008 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} 95.947.595.097.3Large ICCs ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\textrm{ICC}_{X_1} = \ldots = \textrm{ICC}_{X_{25}} = .30$$\end{document} )Single-level regression: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} )500.5000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.015$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.015$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.051}$$\end{document} 94.294.394.196.52000.5000.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.007$$\end{document} 93.193.793.796.110000.5000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 94.896.195.197.1Single-level regression: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2, \ldots , X_{25}$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} )500.040 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 90.593.093.898.42000.0400.0010.0010.0020.00193.995.194.995.710000.0400.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} 94.795.394.895.1Two-level regression (latent covariate): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}^B$$\end{document} )500.5000.001 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.036$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.003$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.047$$\end{document} 93.895.195.499.12000.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.033$$\end{document} 0.002 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.006$$\end{document} 94.293.295.197.310000.5000.000 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.032$$\end{document} 0.003 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} 95.386.294.496.4Note. Bias values larger than ±10% of the true values and coverage rates below 92.5% are printed in bold. Coverage rates above 97.5% are underlined. Monte Carlo SEs ranged from 0.000 to 0.010 for the bias and from 0.3% to 1.6% for the coverage rates. J = number of groups; CD = complete data; FCS-GM = single-level FCS with group means; FCS-AGM = single-level FCS with adjusted group means; FCS-2L = two-level FCS

Results: Study 3a

Due to the large number of results in Study 3a, we focus on the results for the regression coefficients in the single-level regression models and the between-group effects in the latent covariate model, which are summarized in Table 7 (for the full results, see Supplement C). For the single-level analyses, all methods provided estimates with little or no bias, except for FCS-2L, which exhibited a slight downward bias in conditions with large ICCs (ICCs = .30) and a small number of groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=50$$\end{document} ). The coverage rates in the single-level analyses were close to the nominal value of 95% for all methods except CD, which showed coverage rates slightly below the nominal value in conditions with large ICCs (ICCs = .30) and a small number of groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=50$$\end{document} ), and FCS-2L, which showed above-nominal coverage in the same conditions and regardless of the ICCs. For the between-group coefficients in the latent covariate model, CD and FCS-AGM yielded approximately unbiased estimates and near-nominal coverage in all conditions. For FCS-GM, we observed a downward bias in the between-group effects and below-nominal coverage in conditions with small ICCs (ICCs = .10). Finally, for FCS-2L, we found little to no bias, except in one condition with small ICCs (ICCs = .10) and a small number of groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=50$$\end{document} ), but we again observed above-nominal coverage in conditions with a small number of groups ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$J=50$$\end{document} ) regardless of the ICC.Table 8. Study 3b: Bias and coverage rates (in %) of the 95% confidence intervals of the estimated regression coefficients ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} ) in the single- level regressions of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2, \ldots , X_{K}$$\end{document} (cluster-robust SEs) and the estimated between-group coefficient ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}^{B}$$\end{document} ) in the two-level regression of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} (latent covariate)BiasCoverage (%)KTrueCDFCS-GMFCS-AGMFCS-2LCDFCS-GMFCS-AGMFCS-2LSingle-level regression: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} )25 0.5000.003 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.016$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.015$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.051}$$\end{document} 94.993.994.496.135 0.5000.002 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.023$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.020$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.099}$$\end{document} 94.093.694.891.950 0.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.000$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.040$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.035$$\end{document} 95.589.692.9Single-level regression: \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2, \ldots , X_{K}$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}$$\end{document} )25 0.0400.0170.0150.0160.01090.894.193.898.435 0.0290.0150.0170.0160.00891.893.994.6100.050 0.0200.0080.0050.00692.695.697.1Two-level regression (latent covariate): \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\beta _{X_1 X_2}^B$$\end{document} )25 0.5000.014 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.113}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.018$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.125}$$\end{document} 92.894.895.599.335 0.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.001$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.123}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.060}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.240}$$\end{document} 92.094.596.4100.050 0.500 \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$-0.010$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.150}$$\end{document} \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\mathbf{-0.112}$$\end{document} 93.794.996.0Note. Bias values larger than ±10% of the true values and coverage rates below 92.5% are printed in bold. Coverage rates above 97.5% are underlined. Monte Carlo SEs ranged from 0.001 to 0.012 for the bias and from 0.0% to 1.0% for the coverage rates. K = number of variables; CD = complete data; FCS-GM = single-level FCS with group means; FCS-AGM = single-level FCS with adjusted group means; FCS-2L = two-level FCS

Results: Study 3b

For the results of Study 3b, we again focus on the main findings, which are shown in Table 8 (for the full results, see Supplement C). No results are shown for FCS-2L for the condition with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=50$$\end{document} variables, because FCS-2L failed consistently in this condition.4 By contrast, CD and FCS-GM provided results in all replications and all conditions, and FCS-AGM also did so except in the condition with \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=50$$\end{document} , where it failed in four (0.4%) replications. In the single-level regression of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} , CD and FCS-AGM provided results with little bias and near-nominal coverage rates of the 95% confidence intervals. For FCS-GM, the bias was also low, but the coverage was reduced in conditions with many variables ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=50$$\end{document} ). For FCS-2L, the results were biased in all conditions, and coverage was reduced unless the number of variables was only moderate ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=25$$\end{document} ). In the single-level regression of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on all other variables, the results were biased with poor coverage for most methods, including CD. Specifically, CD tended to overestimate the regression coefficient with below-nominal coverage rates of the 95% confidence intervals, particularly when the number of variables was only moderate ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=25$$\end{document} , 35). The results for FCS-GM and FCS-AGM were typically close to CD but with higher (near-nominal) coverage rates. For FCS-2L, the results showed nominally less bias, but the parameter estimates were less similar to CD, and coverage was too high in all conditions. Finally, in the two-level regression of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_1$$\end{document} on \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$X_2$$\end{document} , CD provided results with little bias and near-nominal coverage, whereas the other methods tended to provide poorer results, especially in conditions with many variables ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=35$$\end{document} , 50). For FCS-GM, the results were noticeably biased but with nominal coverage. FCS-AGM also showed bias with nominal coverage, but the bias was the lowest among the MI procedures and limited to conditions with many variables ( \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$K=35$$\end{document} , 50). Finally, FCS-2L showed the strongest bias and above-nominal coverage.

Summary

The results of Studies 3a and 3b indicated that the AGM approach (FCS-AGM) can sometimes provide more accurate results than multilevel MI (FCS-2L) in applications with many variables and a multilevel structure that is not very pronounced. This difference was smaller in Study 3a, where the number of variables was still moderate but were more noticeable in Study 3b, where the number of variables approached the number of groups, and the ICCs were small and heterogeneous. In these situations, the AGM approach tended to perform better than multilevel MI and showed acceptable performance even when multilevel MI could not be used at all.

Discussion

In the present article, we introduced the GM and AGM approach to multilevel MI, which combines single-level imputation methods with (adjusted) group means to accommodate the multilevel structure. After investigating the theoretical properties of the GM and AGM approach, we evaluated them in three simulation studies, which were focused on balanced designs, unbalanced designs, and applications with many variables, where the multilevel structure is not of immediate interest in the substantive analyses.

Overall, our findings indicate that an AGM approach based on single-level MI could be a useful alternative to multilevel MI, particularly in challenging situations where multilevel MI can be less reliable. Specifically, in the less challenging situations, we found that multilevel MI generally performed well and that the AGM approach provided results that were similar to – but also not better than – those in multilevel MI. This was also true in unbalanced data, where the extended specification of the AGM approach provided results similar to multilevel MI. However, in the more challenging situations, the AGM approach often provided substantially more accurate results and was much less prone to computational problems than multilevel MI. By comparison, the GM approach appeared to have poorer theoretical properties and tended to perform worse than the AGM approach in most situations.

Despite these encouraging results, there are multiple limitations worth highlighting. First, we considered only multilevel designs with a two-level structure. The GM and AGM approaches could also be extended to higher-level or non-hierarchical structures. In fact, the AGM approach has already been proposed for handling missing data in cross-classified designs (Grund et al., 2023). However, research on these applications is still scarce, and future research should consider other design types in more detail, including three-level, cross-classified, and multiple-membership designs (see also Wijesuriya et al., 2020; Kadengye et al., 2014). Second, we only considered applications with continuous data. In principle, the GM and AGM approaches could also be combined with single-level imputation methods for categorical or ordinal data (e.g., Austin and van Buuren, 2023), but the use of (adjusted) group means might not be straightforward in these situations (e.g., with unordered categorical data). Third, although our simulations included conditions with many variables, the number of variables that we considered was still lower than in many real-world applications. Recent research has indicated that problems with many variables can be addressed in the FCS approach by combining conventional imputation methods with dimension reduction techniques (Audigier et al., 2016; Costantini et al., 2024; Grund et al., 2024; Howard et al., 2015). However, it is currently an open question whether and how these techniques can be used in multilevel designs. Related to this, we did not consider conditions with many variables and unbalanced designs, which would be an interesting scenario, because the extensions needed to accommodate strongly imbalanced group sizes come at the cost of increasing the number of variables in the imputation model. Finally, throughout the article, we only considered applications that aimed at relatively simple single- and multilevel analyses. This was done deliberately, because we believe that these types of applications are extremely common in practice (e.g., McNeish et al., 2016; Cameron and Miller, 2015) and remain under-explored in the missing data literature. Nonetheless, more complex analyses are also common, for example, when researchers employ multilevel analyses with random slopes and nonlinear effects, and the beneficial properties of the AGM approach may not transfer to these applications (see also Enders et al., 2020; Erler et al., 2016, 2017; Grund et al., 2021; Lüdtke et al., 2020; Keller and Enders, 2023).

To summarize, the aim of the present article was to explore a group-mean-based approach to multilevel MI, in which single-level MI approaches are augmented with (adjusted) group means to accommodate the multilevel structure of the data. Our findings indicate this approach may be a promising method for handling missing data in multilevel designs, which could be particularly useful in challenging situations, in which conventional methods perform less reliably, and in which the multilevel structure does not feature strongly in the intended analyses. We hope that future research will continue in this line of research and that researchers will find this approach useful for handling missing data in multilevel designs.

Author Note

All additional files concerning this manuscript, including code, data, and the supplemental materials, are available at: https://osf.io/mpfzu/. A copy of this manuscript has been posted on the PsyArXiv preprint server: (Grund et al., 2025)

Open Practices Statement

All additional files concerning this article, including the computer code and results of the simulation studies, the computer code and data for the example analysis, and the supplemental materials, are available at: https://osf.io/mpfzu/. None of the simulation studies reported in this article were preregistered.