Frequentist performances of Bayesian prediction intervals for random-effects meta-analysis

TL;DR

This study evaluates the frequentist coverage of Bayesian prediction intervals in meta-analysis, revealing that their accuracy heavily depends on prior choices and sample size, especially when fewer than 10 studies are involved.

Contribution

It provides the first comprehensive simulation assessment of Bayesian prediction intervals' frequentist validity in meta-analysis, highlighting limitations with small sample sizes and various priors.

Findings

Coverage depends on prior distribution choice.

No priors maintained accurate coverage with fewer than 10 studies.

Bayesian prediction intervals can be misleading if used without caution.

Abstract

The prediction interval has been increasingly used in meta-analyses as a useful measure for assessing the magnitude of treatment effect and between-studies heterogeneity. In calculations of the prediction interval, although the Higgins-Thompson-Spiegelhalter method is used most often in practice, it might not have adequate coverage probability for the true treatment effect of a future study under realistic situations. An effective alternative candidate is the Bayesian prediction interval, which has also been widely used in general prediction problems. However, these prediction intervals are constructed based on the Bayesian philosophy, and their frequentist validities are only justified by large-sample approximations even if non-informative priors are adopted. There has been no certain evidence that evaluated their frequentist performances under realistic situations of meta-analyses. In this study, we conducted extensive simulation studies to assess the frequentist coverage performances of Bayesian prediction intervals with 11 non-informative prior distributions under general meta-analysis settings. Through these simulation studies, we found that frequentist coverage performances strongly depended on what prior distributions were adopted. In addition, when the number of studies was smaller than 10, there were no prior distributions that retained accurate frequentist coverage properties. We also illustrated these methods via applications to eight real meta-analysis datasets. The resultant prediction intervals also differed according to the adopted prior distributions. Inaccurate prediction intervals may provide invalid evidence and misleading conclusions. Thus, if frequentist accuracy is required, Bayesian prediction intervals should be used cautiously in practice.