Let's practice what we preach: Planning and interpreting simulation studies with design and analysis of experiments
Hugh Chipman, Derek Bingham

TL;DR
This paper advocates for applying Design and Analysis of Experiments (DAE) methods to improve the planning and interpretation of simulation studies, promoting more rigorous and systematic research practices.
Contribution
It demonstrates how DAE tools like factorial designs and ANOVA can enhance the planning and analysis of simulation studies, including robustness assessments with Taguchi methods.
Findings
DAE methods improve the rigor of simulation study analysis.
Factorial designs facilitate comprehensive exploration of factors.
Taguchi methods assess robustness to population variability.
Abstract
Statisticians recommend the Design and Analysis of Experiments (DAE) for evidence-based research but often use tables to present their own simulation studies. Could DAE do better? We outline how DAE methods can be used to plan and analyze simulation studies. Tools for planning include fishbone diagrams, factorial and fractional factorial designs. Analysis is carried out via ANOVA, main-effect and interaction plots and other DAE tools. We also demonstrate how Taguchi Robust Parameter Design can be used to study the robustness of methods to a variety of uncontrollable population parameters.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsOptimal Experimental Design Methods · Behavioral and Psychological Studies
