Requirements Quality Research Artifacts: Recovery, Analysis, and Management Guideline

TL;DR

This paper addresses the challenge of unavailable research artifacts in requirements quality research by recovering data sets and implementations, analyzing reasons for unavailability, and providing guidelines to promote open science practices.

Contribution

It extends artifact recovery efforts, empirically analyzes unavailability causes, and offers a community-reviewed guideline for open artifact disclosure in requirements quality research.

Findings

Recovered 10 data sets and 7 implementations.

Artifact availability improves over time and with public hosting.

Proposed guideline encourages open science artifact sharing.

Abstract

Requirements quality research, which is dedicated to assessing and improving the quality of requirements specifications, is dependent on research artifacts like data sets (containing information about quality defects) and implementations (automatically detecting and removing these defects). However, recent research exposed that the majority of these research artifacts have become unavailable or have never been disclosed, which inhibits progress in the research domain. In this work, we aim to improve the availability of research artifacts in requirements quality research. To this end, we (1) extend an artifact recovery initiative, (2) empirically evaluate the reasons for artifact unavailability using Bayesian data analysis, and (3) compile a concise guideline for open science artifact disclosure. Our results include 10 recovered data sets and 7 recovered implementations, empirical support for artifact availability improving over time and the positive effect of public hosting services, and a pragmatic artifact management guideline open for community comments. With this work, we hope to encourage and support adherence to open science principles and improve the availability of research artifacts for the requirements research quality community.