A self-assessment Instrument for assessing test automation maturity

TL;DR

This paper presents a validated self-assessment instrument for measuring test automation maturity in organizations, based on a comprehensive literature review and expert evaluation, to improve testing practices.

Contribution

It introduces a new self-assessment tool for test automation maturity, grounded in 15 key areas and validated through expert feedback and cognitive interviews.

Findings

Practices mapped into 15 key areas for assessment

The instrument was validated with expert evaluation

Discussion of response bias in self-assessment tools

Abstract

Test automation is important in software industry but self-assessment instruments for assessing its maturity are not sufficient. The two objectives of this study are to synthesize what an organization should focus to assess its test automation; develop a self-assessment instrument (a survey) for assessing test automation maturity and scientifically evaluate it. We carried out the study in four stages. First, a literature review of 25 sources was conducted. Second, the initial instrument was developed. Third, seven experts from five companies evaluated the initial instrument. Content Validity Index and Cognitive Interview methods were used. Fourth, we revised the developed instrument. Our contributions are as follows: (a) we collected practices mapped into 15 KAs that indicate where an organization should focus to assess its test automation; (b) we developed and evaluated a self-assessment instrument for assessing test automation maturity; (c) we discuss important topics such as response bias that threatens self-assessment instruments. Our results help companies and researchers to understand and improve test automation practices and processes.