Which Requirements Artifact Quality Defects are Automatically Detectable? A Case Study

TL;DR

This study analyzes 166 industry rules for requirements artifacts to estimate that about half can be automatically checked, highlighting the potential and limitations of automation in requirements quality control.

Contribution

It provides the first comprehensive overview of which requirements artifact quality rules are automatable and discusses the main reasons why some rules resist automation.

Findings

53% of rules can be automatically checked with good heuristics

Most automatable rules require only simple techniques

Imprecise rule definitions hinder automation potential

Abstract

[Context] The quality of requirements engineering artifacts, e.g. requirements specifications, is acknowledged to be an important success factor for projects. Therefore, many companies spend significant amounts of money to control the quality of their RE artifacts. To reduce spending and improve the RE artifact quality, methods were proposed that combine manual quality control, i.e. reviews, with automated approaches. [Problem] So far, we have seen various approaches to automatically detect certain aspects in RE artifacts. However, we still lack an overview what can and cannot be automatically detected. [Approach] Starting from an industry guideline for RE artifacts, we classify 166 existing rules for RE artifacts along various categories to discuss the share and the characteristics of those rules that can be automated. For those rules, that cannot be automated, we discuss the main reasons. [Contribution] We estimate that 53% of the 166 rules can be checked automatically either perfectly or with a good heuristic. Most rules need only simple techniques for checking. The main reason why some rules resist automation is due to imprecise definition. [Impact] By giving first estimates and analyses of automatically detectable and not automatically detectable rule violations, we aim to provide an overview of the potential of automated methods in requirements quality control.