A novel robust meta-analysis model using the t distribution for outlier accommodation and detection

TL;DR

This paper introduces a new robust meta-analysis model called tMeta that uses the t distribution to handle and detect outlying studies more effectively than existing methods.

Contribution

The novelty is a simple and adaptive model where the marginal effect size follows a t distribution, avoiding numerical integration and enabling robust outlier handling.

Findings

tMeta performs favorably compared to competitors in the presence of mild outliers.

tMeta remains consistent and robust even when gross outliers are present.

The model uses a fast EM-type algorithm for maximum likelihood estimation.

Abstract

Random effects meta-analysis model is an important tool for integrating results from multiple independent studies. However, the standard model is based on the assumption of normal distributions for both random effects and within-study errors, making it susceptible to outlying studies. Although robust modeling using the t distribution is an appealing idea, the existing work, that explores the use of the t distribution only for random effects, involves complicated numerical integration and numerical optimization. In this article, a novel robust meta-analysis model using the t distribution is proposed (tMeta). The novelty is that the marginal distribution of the effect size in tMeta follows the t distribution, enabling that tMeta can simultaneously accommodate and detect outlying studies in a simple and adaptive manner. A simple and fast EM-type algorithm is developed for maximum likelihood estimation. Due to the mathematical tractability of the t distribution, tMeta frees from numerical integration and allows for efficient optimization. Experiments on real data demonstrate that tMeta is compared favorably with related competitors in situations involving mild outliers. Moreover, in the presence of gross outliers, while related competitors may fail, tMeta continues to perform consistently and robustly.

Figures

Full text

Highlights

What is already know

Random effects model is a popular tool for handling heterogeneity between studies in meta-analysis. However, the standard model is based on the Gaussian assumption and thus is susceptible to outlying studies.

What is new

A novel robust meta-analysis model using student’s t distribution called tMeta is proposed, which is capable of simultaneously accommodating and detecting outlying studies in a simple and adaptive manner. Empirical results show that tMeta is compared favorably with related competitors.

Potential impact for Research Synthesis Methods readers

Compared with related competitors, tMeta frees from numerical integration and allows for efficient optimization, which, to our knowledge, offers the first neat solution to robust meta-analysis modeling using the t distribution. Importantly, tMeta provides a simple but powerful robust meta-analysis tool that can accommodate and detect both mild and gross outliers simultaneously.

Introduction

1

In meta-analyses, the collected studies often exhibit heterogeneity, characterized by greater variation among studies than can be explained by the variation within each study,1 which could result in misleading conclusions about the overall treatment effect.2^,^ 3 The random effects model is a popular tool for handling heterogeneity.4^,^ 5 However, the standard model assumes normal distributions for both random effects and within-study errors (nMeta), making it susceptible to outlying studies.

Outlier detection is a central research area in meta-analysis. Many methods have been developed. For example, a likelihood ratio test was constructed to identify outliers by detecting inflated variance6; a forward search algorithm was developed specifically for this purpose7; several outlier and influence diagnostic procedures in meta-regression models were presented.8 Subsequently, case deletion diagnostics and local influence analysis using multiple perturbation schemes, were investigated.9 Several Bayesian outlier detection measures were also introduced for handling outlying studies in network meta-analysis.10 Another important methodology for dealing with outliers is outlier accommodation or robust estimation, which can down-weight the influence of outliers. For instance, robust functions like Huber’s rho and Tukey’s biweight functions were employed to replace the original non-robust objective function, resulting in robust estimates.11

This article focuses on outlier accommodation and detection simultaneously. Several efforts have been made toward this objective. Non-normal alternatives to normal random effects were investigated, and it was found that the t distribution performs the best (tRE-Meta).12 The shortcoming is that the marginal distribution of $[eqn]$ in tRE-Meta is mathematically intractable. Consequently, numerical integration is required to evaluate the log-likelihood and numerical optimization methods have to be employed for maximum likelihood (ML) estimation. Subsequently, new models where $[eqn]$ has a tractable marginal distribution were presented, including the three parameter symmetric marginal model (SYM-Meta) and the four parameter skew marginal model (SKM-Meta).13 Nevertheless, numerical optimization has still to be employed to obtain ML estimates. As a tractable model, a variant of a two-component mixture model (MIX-Meta) was proposed, with one component modeling standard studies and the other addressing outlying studies.1 In MIX-Meta, the marginal distribution of the observed effect $[eqn]$ is a mixture of two normal distributions. However, MIX-Meta suffers from initialization issues, necessitating multiple runs of the fitting algorithm with different starting values.

The common feature of these methods is that the error terms are assumed to follow the normal distribution. In this article, we break this limitation as the marginal distribution of error term in our proposed model follows the t distribution. It is known that the t distribution includes the normal distribution as a special case when the degrees of freedom $[eqn]$ goes to infinity. This means that tMeta offers greater flexibility and applicability than the conventional normal assumption. The main contributions of this article are as follows.

• The marginal distribution of the effect size $[eqn]$ in tMeta follows the t distribution, enabling it to simultaneously accommodate and detect outliers in a simple and adaptive manner. 1) The t distribution offers an additional robustness tuning parameter which can adaptively down-weight outlying studies. 2) The expected weights follow in proportion to a Beta distribution, providing a useful critical value for outlier detection.
• tMeta provides a simple but powerful robust meta-analysis tool that can accommodate and detect both mild and gross outliers simultaneously. As can be seen from Section 4, 1) tMeta versus SYM-Meta and SKM-Meta. Both the three-parameter SYM-Meta and four-parameter SKM-Meta fail in most of the outlier detection tasks, though they have good performance in outlier accommodation. 2) tMeta versus tRE-Meta and MIX-Meta. While all the three methods can be used to detect mild outliers, tMeta performs the best in outlier accommodation. More importantly, in the presence of gross outliers, both tRE-Meta and MIX-Meta could fail while tMeta still performs satisfactorily.
• Due to its mathematical tractability, tMeta frees from numerical integration and allows for efficient optimization. In contrast, tRE-Meta requires both complicated numerical integration and numerical optimization; SYM-Meta and SKM-Meta involve complex numerical optimization13; MIX-Meta requires multiple runs of the fitting algorithm due to the sensitivity issue of mixture models to initialization.1 To our knowledge, tMeta offers the first neat solution to robust meta-analysis modeling using the t distribution.

The rest of this article is organized as follows. Section 2 reviews some related works. Section 3 proposes our new model tMeta. Section 4 conducts case studies to compare tMeta with several closely related competitors. Section 5 offers a summary of the entire article.

Background

2

In this section, we briefly review some fundamental results concerning the standard model nMeta and Student’s t distribution.

Normal meta-analysis model (nMeta)

2.1

In nMeta, the effect size $[eqn]$ for the i-th study is defined as follows

$$[eqn]$$

where the random effects $[eqn]$ captures heterogeneity across studies and follows $[eqn]$ , the within-study error $[eqn]$ follows $[eqn]$ and they are independent of each other. Here, $[eqn]$ is the overall effect size, $[eqn]$ is the unknown between-study variance and $[eqn]$ is the known within-study variance.

From (1), we have $[eqn]$ . Estimates for the parameters $[eqn]$ and $[eqn]$ can be obtained through maximum likelihood methods.14

Student’s t distribution

2.2

Suppose that a random variable y follows the univariate t distribution $[eqn]$ , with center $[eqn]$ , scale parameter $[eqn]$ , and degrees of freedom $[eqn]$ , then the probability density function (p.d.f.) of y is given by

$$[eqn]$$

where $[eqn]$ is the gamma function and $[eqn]$ is the squared Mahalanobis distance of y from the center $[eqn]$ with respect to $[eqn]$ . If $[eqn]$ , $[eqn]$ ; if $[eqn]$ , Var $[eqn]$ ; and if $[eqn]$ , $[eqn]$ .15

Given a latent weight variable $[eqn]$ distributed as the Gamma distribution $[eqn]$ , y can also be represented hierarchically as a latent variable model as follows15:

$$[eqn]$$

Under model (2), it is easy to obtain the marginal distribution $[eqn]$ by $[eqn]$ 16 and the posterior distribution of $[eqn]$ given y

$$[eqn]$$

Novel robust meta-analysis model

3

In this section, we propose a novel robust meta-analysis model called tMeta. In Section 3.1, we present the model. In Section 3.2, we develop an algorithm for parameter estimation. In Section 3.3 and Section 3.4, we give the details for outlier accommodation and detection in tMeta.

The proposed tMeta model

3.1

Based on the hierarchical representation of the t distribution in Section 2.2, we propose a novel robust random effects meta-analysis model, denoted by tMeta. Its latent variable model can be expressed by

$$[eqn]$$

where, unlike nMeta, the random effects $[eqn]$ and the within-study error $[eqn]$ under tMeta are only conditionally independent; that is, $[eqn]$ and $[eqn]$ are mutually independent given the latent weight $[eqn]$ ; $[eqn]$ is the overall effect size, $[eqn]$ is the unknown between-study variance, $[eqn]$ is the known within-study variance, and the degrees of freedom $[eqn]$ .

According to (3), integrating out the latent weight $[eqn]$ yields the marginal distributions $[eqn]$ and $[eqn]$ . Furthermore, using the property of the normal distribution, it is easy to obtain the conditional distribution of $[eqn]$ given $[eqn]$

$$[eqn]$$

Integrating out the latent weight $[eqn]$ , we obtain an important result that the marginal distribution $[eqn]$ follows a t distribution, that is,

$$[eqn]$$

Note that this result is not available under tRE-Meta model, where the marginal distribution of $[eqn]$ is mathematically intractable. This difference arises because tMeta and tRE-Meta model outliers in distinct ways. In tRE-Meta, outliers are assumed to result solely from extreme variation within studies. By contrast, as shown in (4), tMeta models the importance of a study i at the $[eqn]$ -level by incorporating a latent weight $[eqn]$ associated with $[eqn]$ to reflect the study’s significance. The same $[eqn]$ is then naturally applied to both the between-study effect $[eqn]$ and the within-study error $[eqn]$ , as shown in (3). In other words, outliers in tMeta are assumed to result from extreme variation across both the within-study and between-study levels. This hierarchical modeling framework enables a tractable marginal model for the effect $[eqn]$ .

As a result, the degrees of freedom $[eqn]$ in tMeta can be interpreted as an overall measure of deviation from the nMeta model across both within-study and between-study levels. The two models differ significantly when $[eqn]$ is small but become similar as $[eqn]$ becomes large. Similar overall measures have appeared in the literature; for example, a total correlation parameter has been used to capture overall correlation across both levels in the normal random-effects model.17 Notably, nMeta emerges as a special case of tMeta in the limit, as the t distribution $[eqn]$ approaches the normal distribution $[eqn]$ as $[eqn]$ .

Probability distributions

3.1.1

From tMeta model (3), it is easy to obtain the following probability distributions

$$[eqn]$$

where

$$[eqn]$$

is the squared Mahalanobis distance of $[eqn]$ from the overall effect size $[eqn]$ . It is clear that all the probability distributions under tMeta, including the marginal distributions of $[eqn]$ , $[eqn]$ and $[eqn]$ given in Section 3.1, are well-known and tractable.

Robust meta-regression with covariates

3.1.2

When several covariates are involved, the model (3) can be extended to a more general model,

$$[eqn]$$

where $[eqn]$ represents p-dimensional vector of covariates, $[eqn]$ is the p-dimensional regression coefficients; the random variables $[eqn]$ and $[eqn]$ and the other parameters $[eqn]$ and $[eqn]$ are similar as those in tMeta (3). Under this model, we have $[eqn]$ .

Maximum likelihood estimation

3.2

In this section, we develop estimation algorithms for obtaining the ML estimates of the parameters $[eqn]$ in the tMeta model. Given the effect size vector $[eqn]$ , from (3) the observed data log-likelihood function $[eqn]$ is (up to a constant),

$$[eqn]$$

The maximization of $[eqn]$ in (8) can be obtained by standard numerical optimizers. However, we shall propose an EM-type algorithm to obtain the ML estimate $[eqn]$ because of its simplicity and stability.15 From (6), the required conditional expectation in the E-step can be obtained as

$$[eqn]$$

The details about the development of this algorithm can be found in Section A.1 of the Appendix.

Outlier accommodation

3.3

Adaptive outlier accommodation

3.3.1

Looking at (9), (A.2), and (A.3), the following can be observed.

• When the data contain no outliers and the $[eqn]$ ’s come from nMeta, $[eqn]$ is expected to take on large values. This causes all the weights $[eqn]$ in (9) to be close to 1. Consequently, (A.2) and (A.3) would degenerate to those of nMeta, and hence tMeta adaptively degenerates to nMeta in this case.
• In the presence of outliers, $[eqn]$ is expected to take on small values, and the outlying study $[eqn]$ would have a much greater squared Mahalanobis distance $[eqn]$ compared with non-outliers, causing the outlier’s $[eqn]$ in (9) to be much smaller than those of non-outliers. Consequently, the impact of outliers on the estimators in (A.2) and (A.3) is substantially reduced, allowing tMeta to yield robust estimates. In summary, the degrees of freedom $[eqn]$ is a robustness tuning parameter that adapts according to the presence of outliers in the data.

Breakdown point

3.3.2

In statistics, the robustness of estimators is assessed by breakdown points, which are the proportion of arbitrarily large outlying observations an estimator can tolerate before giving an incorrect result. The following Proposition 1 gives the breakdown point of tMeta.Proposition 1.The upper bound of the breakdown point of tMeta is $[eqn]$ . Proof.As proved by Dümbgen and Tyler,18 the upper bound of the breakdown point of the d-dimensional multivariate t distribution is $[eqn]$ . For tMeta, the dimension of t-distributed $[eqn]$ is $[eqn]$ and hence the upper bound of tMeta is given by $[eqn]$ . This completes the proof.

In our implementation, we restrict $[eqn]$ . Proposition 1 shows that tMeta is a highly robust method as its breakdown point could be close to 50% under this restriction.

Outlier detection

3.4

Similar to that in multivariate t and matrix-variate t distributions,19^–^ 21 the expected weight $[eqn]$ in tMeta given by (9) can be used as outlier indicator. Let

$$[eqn]$$

The following Proposition 2 gives the details.Proposition 2.Assume that the study $[eqn]$ follow tMeta model (3). Given the ML estimate $[eqn]$ , we have, when the estimate $[eqn]$ ,

$$[eqn]$$

and when $[eqn]$ ,

$$[eqn]$$

Proof.The proof can be found in Section A.2 of the Appendix.

Proposition 2 shows that when the estimate $[eqn]$ , the average of all $[eqn]$ ’s equals to 1. In other words, the study with $[eqn]$ much smaller than 1 (i.e., $[eqn]$ much smaller than $[eqn]$ ) or close to 0 can be considered as an outlier. When $[eqn]$ , our experience reveals that $[eqn]$ may be slightly greater than $[eqn]$ .

In practice, a critical value is needed to judge whether a study is an outlier or not. The following Proposition 3 does this task. Let $[eqn]$ and $[eqn]$ stand for the F distribution and Beta distribution with parameters a and b, respectively. The $[eqn]$ quantile of $[eqn]$ is denoted by $[eqn]$ .Proposition 3.Suppose that the study set $[eqn]$ follow tMeta model (3). Then we have that the Mahalanobis distance $[eqn]$ . Given the ML estimate $[eqn]$ , the weights $[eqn]$ converge in distribution to $[eqn]$ as the study sample size N approaches infinity. Therefore, at a significance level of $[eqn]$ , the i-th study with $[eqn]$ could be identified as an outlier. Proof.This is a special case with dimension $[eqn]$ of the result on the d-dimensional multivariate t distribution proved by Wang and Fun.19 This completes the proof.

Results

4

In this section, we compare the performance of our proposed tMeta with five closely related methods: nMeta, tRE-Meta, MIX-Meta, SYM-Meta and SKM-Meta using four benchmark real-world datasets. For tMeta, the iteration stops when the relative change in the objective function $[eqn]$ (|1- $[eqn]$ ) is smaller than the given threshold $[eqn]$ or the number of iterations exceeds $[eqn]$ . For nMeta, tRE-Meta, and MIX-Meta, we use the R codes available from https://cran.r-project.org/web/packages/metaplus/. In addition, we use the default setting for MIX-Meta, i.e., 20 initializations. The code for SYM-Meta and SKM-Meta can be found from the supplementary materials by Baker and Jackson.13

To perform outlier detection for tMeta, we utilize the critical value provided in Proposition 3 and set the significance level $[eqn]$ . For better visualization, we equivalently plot the inverse of $[eqn]$ . That is, the study with $[eqn]$ is identified as an outlier for tMeta. For MIX-Meta, we use the empirical threshold 0.9 as suggested by Beath,1 which represents the posterior probability that a study belongs to the outlying component. For SYM-Meta and SKM-Meta, we adopt the p-value method specially developed for both models by Baker and Jackson.13 Since tRE-Meta lacks guidelines for setting the threshold, we follow the empirical approach by Baker and Jackson,12 treating studies with very small values of the relative weight $[eqn]$ , or equivalently, very large values of $[eqn]$ as outliers, where $[eqn]$ and $[eqn]$ are the weights under tRE-Meta and nMeta, respectively.

To compare the computational efficiency, we will report their total CPU time consumed by various methods, which is sum of the time used for training model and that for detecting outliers. For tMeta and MIX-Meta, outlier detection is a byproduct of the model training and incurs no additional time cost. However, tRE-Meta, SYM-Meta and SKM-Meta require additional time cost for outlier detection. To be specific, tRE-Meta requires numerical methods to compute $[eqn]$ while SYM-Meta and SKM-Meta necessitate additional efforts to implement the p-value method.

Intravenous magnesium

4.1

The Mag dataset22 comprises 16 studies. As can be seen from the forest plot shown in Figure 1(a), it looks difficult to visually identify which study is an outlier except that study 16 seems different from others due to its relatively large $[eqn]$ value and low $[eqn]$ . Previous researches1^,^ 6 have analyzed this dataset and found no outliers. Below we perform outlier detection with various methods.Figure 1 Top row: forest plots on two datasets: (a) Mag and (b) Hipfrac, where each effect size $[eqn]$ and 95% confidence interval are shown as circle and solid line, respectively. Bottom row: evolement of log-likelihood of $[eqn]$ versus number of iterations: (c) Mag and (d) Hipfrac.

We fit all the six methods on the Mag dataset. Table 1 collects the results. The results in Table 1 show all the six methods yield similar performance. This means that all the five methods tRE-Meta, MIX-Meta, SYM-Meta, SKM-Meta and tMeta could degrade to nMeta. Nevertheless, among the five robust methods, tMeta is computationally the most efficient while tRE-Meta and MIX-Meta require much more time. Figure 1(c) shows the evolvement of log-likelihood $[eqn]$ versus number of iterations when fitting tMeta. It can be seen from Figure 1(c) that tMeta converges within 7 iterations on this dataset.Table 1 Results of parameter estimates, negative log-likelihood, and CPU time (in seconds) by various methods on Mag datasetMethods $[eqn]$

$[eqn]$

$[eqn]$

• $[eqn]$ TimenMeta $[eqn]$ 0.7460.504—19.685— tRE-Meta $[eqn]$ 0.7460.504inf19.6853.0MIX-Meta $[eqn]$ 0.7460.504—19.68532.4SYM-Meta $[eqn]$ 0.7460.504—19.6850.3SKM-Meta $[eqn]$ 0.7460.504—19.6850.3 tMeta $[eqn]$ 0.7460.504inf19.6850.05 Note: ‘—’ indicates that a method does not have corresponding results.

Figure 2 shows the results of detecting outliers by the five methods. It can be seen from Figure 2 that all the five methods suggest no outliers for Mag dataset. This finding is consistent with that by Beath.1 Figure 2 Results on outlier detection by various methods on Mag dataset: (a) tMeta; (b) tRE-Meta; (c) MIX-Meta; (d) SYM-Meta; (e) SKM-Meta. The marker solid point $[eqn]$ in blue represents normal studies judged by a method.

Hipfrac dataset

4.2

The Hipfrac dataset23 contains 17 studies, collected from an investigation on the magnitude and duration of excess mortality after hip fracture among older men. Figure 1(b) shows the forest plot, from which it seems hard to identify which study is an outlier. Below we perform outlier analysis with various methods.

We fit all the six methods on the Hipfrac dataset. Table 2 summarizes the results. The results in Table 2 show that tMeta and SKM-Meta obtain significantly better BIC than the other methods and SKM-Meta wins by a narrow margin. In terms of computational efficiency among the five robust methods, tMeta is the fastest while tRE-Meta and MIX-Meta are the slowest runners. Figure 1(d) shows the evolution of log-likelihood $[eqn]$ versus number of iterations when fitting tMeta. It can be seen from Figure 1(d) that tMeta converges within 6 iterations on this dataset.Table 2 Results of parameter estimates, negative log-likelihood, BIC, and CPU time (in seconds) by various methods on Hipfrac datasetMethods $[eqn]$

$[eqn]$

$[eqn]$

• $[eqn]$ BICTimenMeta1.3570.260—8.49822.661— tRE-Meta1.2510.0130.5826.57521.64979.2MIX-Meta1.2520.000—4.50720.347364.1SYM-Meta1.2200.074—5.67019.8400.222SKM-Meta1.2020.063—1.439 14.212 0.2 tMeta1.2520.0001.8713.70015.8990.03 Note: The best method is shown in boldface. ‘—’ indicates that a method does not have corresponding results.

Figure 3 shows the results of detecting outliers by the five methods. It can be seen from Figure 3 that both tMeta and tRE-Meta identify study 17 as an outlier. This result is consistent with that obtained by Lin et al.2 In contrast, MIX-Meta identifies one more outlier: study 9, while SYM-Meta and SKM-Meta fail completely.Figure 3 Results on outlier detection by various methods on Hipfrac dataset: (a) tMeta; (b) tRE-Meta; (c) MIX-Meta; (d) SYM-Meta; (e) SKM-Meta. The vertical line indicates the critical value for tMeta (red) and the threshold 0.9 (magenta) for MIX-Meta. The vertical line indicates the critical value for tMeta and the threshold 0.9 for MIX-Meta. The marker solid point $[eqn]$ in blue represents normal studies judged by a method. Star ‘’ signals outlying studies, with red for tMeta and magenta for the other methods*.

It is interesting to make a further comparison between tMeta and tRE-Meta. From Figure 3(a), it can be seen that tMeta detects that study 9 as being close to an outlier, while it is not the case by tRE-Meta from Figure 3(b). In fact, Lin et al.2 have considered study 9 as a potential outlier and perform a sensitivity analysis by removing this study. As a result, they found that study 9 is not influential. Therefore, the result by tMeta is well match that obtained by Lin et al.2

Fluoride toothpaste

4.3

This dataset contains 70 studies, obtained from an evaluation of fluoride’s efficacy in preventing childhood dental caries.24 The effect size $[eqn]$ denotes the difference between control and treatment groups, with negative values signifying significant therapeutic effects.

Previous works1^,^ 6^,^ 12 have concluded that there exist three outliers in this dataset: study 63, study 50 and study 38. Contrarily, the analysis with SKM-Meta suggests no outliers in the dataset.13 To better examine the outlier detection performance by various methods, we shall perform two experiments in this section. In the first experiment of Section 4.3, we use the original dataset (Flu). In the second experiment of Section 4.3, we add the original dataset with one more artificial outlier. The resulting dataset is called modified Flu for clarity.

Original Flu

4.3.1

Figure 4(a) shows the forest plot of the original dataset Flu. It can be observed from Figure 4(a) that studies 38, 50, and 63 look like abnormal. We then perform further analysis to identify outliers.Figure 4 Top row: forest plots on the fluoride toothpaste dataset: (a) Flu and (b) modified Flu, where each effect size $[eqn]$ and 95% confidence interval are shown as circle and solid line, respectively. Bottom row: evolement of log-likelihood of $[eqn]$ versus number of iterations: (c) Flu and (d) modified Flu.

We fit all the six methods on Flu. Table 3 summarizes the results. The results in Table 3 show that tMeta, SYM-Meta and SKM-Meta obtain substantially better BIC than the other methods and SKM-Meta is again the best. Among the five robust methods, tMeta is the most computationally efficient while tRE-Meta and MIX-Meta are the most inefficient. Figure 4(c) shows the evolution of log-likelihood $[eqn]$ versus number of iterations when fitting tMeta. It can be seen from Figure 4(c) that tMeta converges within 18 iterations on this dataset.Table 3 Results by various methods on the original and modified fluoride toothpaste dataset, including parameter estimates, negative log-likelihood, BIC, and CPU time (in seconds)Methods $[eqn]$

$[eqn]$

$[eqn]$

• $[eqn]$ BICTimeOriginal FlunMeta $[eqn]$ 0.3000.119—1.23310.963— tRE-Meta $[eqn]$ 0.2800.0491.158 $[eqn]$ 13.121 $[eqn]$ 13.49764.0MIX-Meta $[eqn]$ 0.2810.090— $[eqn]$ 14.636 $[eqn]$ 12.27727.0SYM-Meta $[eqn]$ 0.2820.092— $[eqn]$ 17.148 $[eqn]$ 21.5510.4SKM-Meta $[eqn]$ 0.2730.081— $[eqn]$ 21.914 $[eqn]$ 26.834 0.6 tMeta $[eqn]$ 0.2820.0512.754 $[eqn]$ 18.283 $[eqn]$ 23.8200.05Modified FlunMeta $[eqn]$ 0.2970.139—15.76040.046— tRE-Meta $[eqn]$ 0.2790.0471.023 $[eqn]$ 7.774 $[eqn]$ 2.76159.2MIX-Meta $[eqn]$ 0.2800.088— $[eqn]$ 10.062 $[eqn]$ 3.07226.9SYM-Meta $[eqn]$ 0.2820.092— $[eqn]$ 12.399 $[eqn]$ 12.0100.7SKM-Meta $[eqn]$ 0.2770.088— $[eqn]$ 13.144 $[eqn]$ 9.2380.6 tMeta $[eqn]$ 0.2810.0472.367 $[eqn]$ 13.791 $[eqn]$ 14.794 0.05 Note: The best method is shown in boldface. ‘—’ indicates that a method does not have corresponding results.

The top row in Figure 5 shows the results of detecting outliers by the five methods. It can be seen from Figure 5 that tMeta, tRE-Meta and MIX-Meta all identify three studies: 63, 50, 38. This means that the result by tMeta is consistent with those in previous works.1^,^ 6^,^ 12 In contrast, SYM-Meta only detects the most abnormal study 63 as one outlier while SYM-Meta identify no outlier.

Modified Flu

4.3.2

In the modified Flu, the outlier (study 71) is introduced as follows. The effect size $[eqn]$ is generated from the uniform distribution U on the interval $[eqn]$ , i.e., $[eqn]$ . We set its within-study variance $[eqn]$ . Figure 4(b) shows the forest plot of modified Flu, from which it can be seen that the newly added study 71 looks like a mild outlier as it is very different from all the other studies.

Table 3 summarizes the results by six methods. The results in Table 3 show that tMeta yields the best BIC on this dataset, which is then followed by SYM-Meta, and SKM-Meta is the third best. Among the five robust methods, tMeta is again the best performer in computational efficiency while tRE-Meta and MIX-Meta are still the most inefficient. Figure 4(d) shows the evolution of log-likelihood $[eqn]$ versus number of iterations when fitting tMeta. It can be seen from Figure 4(d) that tMeta converges within 19 iterations on this dataset.

The bottom row in Figure 5 shows the results of detecting outliers by the five methods. It can be seen from the bottom row of Figure 5 that tMeta, tRE-Meta and MIX-Meta successfully identify four outliers: 71, 63, 50, 38. In contrast, SYM-Meta fails to detect any outlier, while SKM-Meta can detect the newly added study 71.Figure 5 Results on outlier detection by various methods on fluoride toothpaste dataset. Top row: the original dataset; Bottom row: the modified dataset. (a), (f) tMeta; (b), (g) tRE-Meta; (c), (h) MIX-Meta; (d), (i) SYM-Meta; (e), (j) SKM-Meta. The vertical line indicates the critical value for tMeta and the threshold 0.9 for MIX-Meta. The marker solid point $[eqn]$ in blue represents normal studies judged by a method. Star ‘’ signals outlying studies, with red for tMeta and magenta for the other methods*.

CDP-choline

4.4

The CDP-choline dataset25 is obtained by exploring the cytidinediphosphocholine analysis in cognitive and behavioural disorders associated with chronic brain diseases in the elderly. The sample size is $[eqn]$ .

Previous analyses1^,^ 6^,^ 12 have concluded that there is one outlier in this dataset: study 8. Like Section 4.3, we perform two experiments. In the first experiment of Section 4.4, we use the original dataset (CDP). In the second experiment of Section 4.4, we modify CDP so that it contains more outliers, which is denoted by modified CDP for clarity.

Original CDP

4.4.1

Figure 6(a) shows the forest plot of the original CDP. It can be observed from Figure 6(a) that study 8 looks like abnormal. We then perform further analysis to identify outliers.Figure 6 Top row: forest plots on CDP-choline dataset: (a) original dataset; (b) modified dataset. Bottom row: evolement of log-likelihood of $[eqn]$ versus number of iterations: (c) original dataset and (d) modified dataset. Figure 7 Results on outlier detection by various methods on CDP-choline dataset. Top row: the original dataset; Bottom row: the modified dataset. (a), (f) tMeta; (b), (g) tRE-Meta; (c), (h) MIX-Meta; (d), (i) SYM-Meta; (e), (j) SKM-Meta. The vertical line indicates the critical value for tMeta and the threshold 0.9 for MIX-Meta. The marker solid point $[eqn]$ in blue represents normal studies judged by a method. Star ‘’ signals outlying studies, with red for tMeta and magenta for the other methods*.

We fit all the six methods on CDP. Table 4 summarizes the results. The results in Table 4 show that tMeta, SYM-Meta and SKM-Meta obtain significantly better BIC than the other methods and SKM-Meta is again the best. In terms of computational efficiency among the five robust methods, tMeta is the most efficient while tRE-Meta and MIX-Meta are the slowest runners. Figure 6(c) shows the evolution of log-likelihood $[eqn]$ versus number of iterations when fitting tMeta. It can be seen from Figure 6(c) that tMeta converges within 10 iterations on this dataset.Table 4 Results by various methods on the CDP-choline dataset, including parameter estimates, negative log-likelihood, BIC, and CPU time (in seconds)Methods $[eqn]$

$[eqn]$

$[eqn]$

• $[eqn]$ BICTimeOriginal CDPnMeta0.3890.383—8.19921.002— tRE-Meta0.1950.0060.4944.05815.02424.7MIX-Meta0.1911.777—3.00715.22547SYM-Meta0.1940.000—2.84712.6020.14SKM-Meta0.1930.000—1.403 12.016 0.2 tMeta0.1870.0002.3803.37713.6620.03Modified CDPnMeta5.87917.126—46.85598.506— tRE-Meta0.1930.0020.27313.768 34.729 65.9MIX-Meta5.8792.455—46.855103.30219.5SYM-Meta5.88017.117—46.855100.9040.3SKM-Meta0.4840.711—21.62252.8360.4 tMeta0.2000.1151.00017.08141.3550.03 Note: The best method is shown in boldface. ‘—’ indicates that a method does not have corresponding results.

The top row in Figure 7 shows the results of detecting outliers by the five methods. It can be seen from Figure 5 that all the five methods successfully identify study 8 as an outlier. This means that the results by tMeta, SYM-Meta and SKM-Meta are consistent with those in previous works.1^,^ 6^,^ 12

Modified CDP

4.4.2

In the modified CDP, we make two modifications: (i) add one outlier, namely study 11, which is set as $[eqn]$ ; (ii) set $[eqn]$ . Figure 6(b) shows the forest plot of modified CDP, from which it can be seen that the newly added study 11 is a gross outlier as it is extremely different from all the other studies and study 8 is a mild outlier but now it has a smaller within-study variance than that in the original CDP.

Table 4 summarizes the results by six methods. The results in Table 4 show that tRE-Meta and tMeta have better BIC than the other methods on this dataset and tRE-Meta is the best. In terms of computational efficiency among the five robust methods, tMeta is still the most efficient while tRE-Meta and MIX-Meta demand the most time. Figure 6(d) shows the evolution of log-likelihood $[eqn]$ versus number of iterations when fitting tMeta. It can be seen from Figure 6(d) that tMeta requires 29 iterations to converge on this dataset.

The bottom row in Figure 7 shows the results of detecting outliers by the five methods. It can be seen that tMeta performs reliably as it successfully detects the two outliers: study 11, 8. In contrast, MIX-Meta and SKM-Meta fail to detect the most extreme study 11. tRE-Meta and SYM-Meta can identify study 11 but they fail to detect study 8.

Conclusion

5

For outlier accommodation and detection simultaneously, in this article we propose a novel robust meta-analysis model using student’s t distribution, namely tMeta. tMeta can be expressed as a hierarchical latent variable model while the marginal distribution of the effect size $[eqn]$ follows a tractable t distribution. To obtain the ML estimates of the parameters, we develop an ECME algorithm, which is computationally much more efficient than related methods as shown in our experiments. Empirical results on real datasets show that tMeta not only improves the robustness of nMeta as expected but also is compared favorably with closely related competitors in that it can provide the best performance for outlier accommodation and detection simultaneously, for both mild and gross outliers.

The experiment results show that SKM-Meta on some datasets yields better performance in outlier accommodation. For future work it would be interesting to extend tMeta using the skew-t distribution for further accommodating skewed data.

Supporting information

Wang et al. supplementary materialWang et al. supplementary material