Proposal of an alternative way of reporting the results of comparative simulation studies

TL;DR

This paper proposes a new method for reporting simulation study results that improves clarity and provides more detailed insights for researchers.

Contribution

The paper introduces an Analysis Plan with Traceability Tables and Variability Sets to better present simulation results.

Findings

The proposed method provides more information than existing alternatives for method researchers.

Traceability Tables and Variability Sets help visualize the behavior and variability of analytical approaches.

The R Shiny application supports the implementation of the proposed reporting method.

Abstract

Monte Carlo simulation studies allow testing multiple experimental conditions, whose results are often difficult to communicate and visualize to their full extent. Some researchers have proposed alternatives to address this issue, highlighting its relevance. This article develops a new way of observing, analyzing, and presenting the results of simulation experiments and is explained step by step with an example.

A criterion is proposed to decide which results could be averaged and which results should not be averaged. It is also indicated how to construct Traceability Tables. These tables will show the behavior of the different analytical approaches studied under the chosen conditions and their variability under the averaged conditions. A way of observing the influence of the manipulated variables on the performance of the set of analysis approaches studied is also developed, Variability Set. Finally, a way of exposing the procedures that have the best performance in a particular condition is suggested.

This Analysis Plan for reporting the results of simulation studies provides more information than existing alternative procedures, provides valuable information for method researchers, and specifies to applied researchers which statistic they should use in a particular condition. An R Shiny application is provided.

Figures