The DEVStone Metric: Performance Analysis of DEVS Simulation Engines

TL;DR

The paper introduces the DEVStone metric, a standardized performance measure for DEVS simulation engines, enabling objective comparison across different simulators using a synthetic benchmark.

Contribution

It defines a new performance metric based on the DEVStone benchmark, facilitating consistent evaluation and comparison of DEVS simulation engines.

Findings

The DEVStone metric provides a quantitative basis for comparing DEVS simulators.

Using the metric, the paper compares popular DEVS simulators and assesses feature impacts.

The metric captures diverse model characteristics to ensure comprehensive performance evaluation.

Abstract

The DEVStone benchmark allows us to evaluate the performance of discrete-event simulators based on the DEVS formalism. It provides model sets with different characteristics, enabling the analysis of specific issues of simulation engines. However, this heterogeneity hinders the comparison of the results among studies, as the results obtained on each research work depend on the chosen subset of DEVStone models. We define the DEVStone metric based on the DEVStone synthetic benchmark and provide a mechanism for specifying objective ratings for DEVS-based simulators. This metric corresponds to the average number of times that a simulator can execute a selection of 12 DEVStone models in one minute. The variety of the chosen models ensures we measure different particularities provided by DEVStone. The proposed metric allows us to compare various simulators and to assess the impact of new features on their performance. We use the DEVStone metric to compare some popular DEVS-based simulators.