Designing a Data Science simulation with MERITS: A Primer

TL;DR

This paper introduces the MERITS framework, inspired by PCS, to guide the design of high-quality, trustworthy Data Science simulations through six key principles, a conceptual recipe, guidelines, and a case study.

Contribution

It proposes a comprehensive set of six MERITS principles for designing high-quality Data Science simulations, along with a conceptual framework, guidelines, and a practical case study.

Findings

The MERITS framework effectively guides simulation design.

The analogy with cooking helps conceptualize simulation components.

The case study demonstrates practical application of the framework.

Abstract

Simulations play a crucial role in the modern scientific process. Yet despite (or due to) this ubiquity, the Data Science community shares neither a comprehensive definition for a "high-quality" study nor a consolidated guide to designing one. Inspired by the Predictability-Computability-Stability (PCS) framework for 'veridical' Data Science, we propose six MERITS that a simulation study should satisfy. (Modularity and Efficiency support the computability of a study, encouraging clean and flexible implementation. Realism and Stability address the conceptualization of the research problem: How well does a study predict reality, such that its conclusions generalize to new data/contexts? Finally, Intuitiveness and Transparency encourage good communication and trustworthiness of study design and results.) Drawing an analogy between simulation and cooking, we moreover offer (a) a conceptual framework for thinking about the anatomy of a simulation 'recipe'; (b) a baker's dozen in guidelines to aid the Data Science practitioner in designing one; and (c) a case study demonstrating the practical utility of our framework by using it to autopsy a preexisting simulation study. With this "PCS primer" for high-quality Data Science simulation, we seek to distill and enrich the best practices of simulation across disciplines into a cohesive recipe for trustworthy, veridical Data Science.