Exploring Consequences of Simulation Design for Apparent Performance of Statistical Methods. 1: Results from simulations with constant sample sizes

TL;DR

This paper examines how simulation design choices, specifically constant sample sizes, influence the perceived performance of statistical methods in meta-analysis of log-odds-ratios, highlighting the importance of simulation setup.

Contribution

It demonstrates that the distribution choices for sample sizes and control probabilities significantly impact simulation-based evaluations of statistical methods.

Findings

Simulation design choices affect method performance conclusions

Constant sample sizes yield different results than variable distributions

Future work will explore normal and uniform sample size distributions

Abstract

Contemporary statistical publications rely on simulation to evaluate performance of new methods and compare them with established methods. In the context of meta-analysis of log-odds-ratios, we investigate how the ways in which simulations are implemented affect such conclusions. Choices of distributions for sample sizes and/or control probabilities considerably affect conclusions about statistical methods. Here we report on the results for constant sample sizes. Our two subsequent publications will cover normally and uniformly distributed sample sizes.