Replicability Study: Corpora For Understanding Simulink Models & Projects

TL;DR

This study evaluates the reproducibility of previous Simulink model research using the SLNET corpus, compares open-source and proprietary models, and provides insights into the generalizability of earlier findings in model-based development.

Contribution

It replicates prior analyses with SLNET, introduces a heuristic for identifying code-generating models, and assesses the similarity between open-source and proprietary Simulink models.

Findings

SLNET confirms and contradicts earlier findings.

Open-source models follow good practices and are comparable to proprietary models.

Collected 208 repositories with over 9,000 commits for model evolution studies.

Abstract

Background: Empirical studies on widely used model-based development tools such as MATLAB/Simulink are limited despite the tools' importance in various industries. Aims: The aim of this paper is to investigate the reproducibility of previous empirical studies that used Simulink model corpora and to evaluate the generalizability of their results to a newer and larger corpus, including a comparison with proprietary models. Method: The study reviews methodologies and data sources employed in prior Simulink model studies and replicates the previous analysis using SLNET. In addition, we propose a heuristic for determining code-generating Simulink models and assess the open-source models' similarity to proprietary models. Results: Our analysis of SLNET confirms and contradicts earlier findings and highlights its potential as a valuable resource for model-based development research. We found that open-source Simulink models follow good modeling practices and contain models comparable in size and properties to proprietary models. We also collected and distribute 208 git repositories with over 9k commits, facilitating studies on model evolution. Conclusions: The replication study offers actionable insights and lessons learned from the reproduction process, including valuable information on the generalizability of research findings based on earlier open-source corpora to the newer and larger SLNET corpus. The study sheds light on noteworthy attributes of SLNET, which is self-contained and redistributable.