What Makes Agile Test Artifacts Useful? An Activity-Based Quality Model from a Practitioners' Perspective

TL;DR

This paper investigates the design and quality factors of Agile test artifacts through an industry survey, identifying key challenges and proposing a quality model to improve their usefulness in practice.

Contribution

It presents the first empirical activity-based quality model for Agile test artifacts, based on practitioner insights and identifying critical quality factors and challenges.

Findings

Nine challenges in using test artifacts identified

Six test artifacts analyzed for quality factors

Language and traceability are common issues

Abstract

Background: The artifacts used in Agile software testing and the reasons why these artifacts are used are fairly well-understood. However, empirical research on how Agile test artifacts are eventually designed in practice and which quality factors make them useful for software testing remains sparse. Aims: Our objective is two-fold. First, we identify current challenges in using test artifacts to understand why certain quality factors are considered good or bad. Second, we build an Activity-Based Artifact Quality Model that describes what Agile test artifacts should look like. Method: We conduct an industrial survey with 18 practitioners from 12 companies operating in seven different domains. Results: Our analysis reveals nine challenges and 16 factors describing the quality of six test artifacts from the perspective of Agile testers. Interestingly, we observed mostly challenges regarding language and traceability, which are well-known to occur in non-Agile projects. Conclusions: Although Agile software testing is becoming the norm, we still have little confidence about general do's and don'ts going beyond conventional wisdom. This study is the first to distill a list of quality factors deemed important to what can be considered as useful test artifacts.