Let's practice what we preach: Planning and interpreting simulation studies with design and analysis of experiments

TL;DR

This paper advocates for applying Design and Analysis of Experiments (DAE) methods to improve the planning and interpretation of simulation studies, promoting more rigorous and systematic research practices.

Contribution

It demonstrates how DAE tools like factorial designs and ANOVA can enhance the planning and analysis of simulation studies, including robustness assessments with Taguchi methods.

Findings

DAE methods improve the rigor of simulation study analysis.

Factorial designs facilitate comprehensive exploration of factors.

Taguchi methods assess robustness to population variability.

Abstract

Statisticians recommend the Design and Analysis of Experiments (DAE) for evidence-based research but often use tables to present their own simulation studies. Could DAE do better? We outline how DAE methods can be used to plan and analyze simulation studies. Tools for planning include fishbone diagrams, factorial and fractional factorial designs. Analysis is carried out via ANOVA, main-effect and interaction plots and other DAE tools. We also demonstrate how Taguchi Robust Parameter Design can be used to study the robustness of methods to a variety of uncontrollable population parameters.