Quality Guidelines for Research Artifacts in Model-Driven Engineering

TL;DR

This paper presents tailored quality guidelines for sharing research artifacts in Model-Driven Engineering, developed through analysis and community feedback, to enhance reproducibility and reuse in MDE research.

Contribution

It introduces a set of discipline-specific artifact sharing guidelines for MDE, based on systematic analysis and community survey insights.

Findings

Guidelines were positively evaluated by over 92% of participants.

Community feedback prioritized essential guidelines for artifact sharing.

The full set of guidelines and artifacts are openly available online.

Abstract

Sharing research artifacts is known to help people to build upon existing knowledge, adopt novel contributions in practice, and increase the chances of papers receiving attention. In Model-Driven Engineering (MDE), openly providing research artifacts plays a key role, even more so as the community targets a broader use of AI techniques, which can only become feasible if large open datasets and confidence measures for their quality are available. However, the current lack of common discipline-specific guidelines for research data sharing opens the opportunity for misunderstandings about the true potential of research artifacts and subjective expectations regarding artifact quality. To address this issue, we introduce a set of guidelines for artifact sharing specifically tailored to MDE research. To design this guidelines set, we systematically analyzed general-purpose artifact sharing practices of major computer science venues and tailored them to the MDE domain. Subsequently, we conducted an online survey with 90 researchers and practitioners with expertise in MDE. We investigated our participants' experiences in developing and sharing artifacts in MDE research and the challenges encountered while doing so. We then asked them to prioritize each of our guidelines as essential, desirable, or unnecessary. Finally, we asked them to evaluate our guidelines with respect to clarity, completeness, and relevance. In each of these dimensions, our guidelines were assessed positively by more than 92\% of the participants. To foster the reproducibility and reusability of our results, we make the full set of generated artifacts available in an open repository at \texttt{\url{https://mdeartifacts.github.io/}}.