Industrially Applicable System Regression Test Prioritization in Production Automation

TL;DR

This paper presents an industrially applicable method for prioritizing system regression tests in automated production systems, using past runtime data and impact analysis to efficiently identify regressions early and reduce testing time under high-pressure conditions.

Contribution

It introduces a novel approach that leverages historical test data and change impact analysis for test case prioritization in production automation systems, addressing a gap in automated support.

Findings

Promising results in industrial case study

Effective early regression detection

Time savings in regression testing process

Abstract

When changes are performed on an automated production system (aPS), new faults can be accidentally introduced in the system, which are called regressions. A common method for finding these faults is regression testing. In most cases, this regression testing process is performed under high time pressure and on-site in a very uncomfortable environment. Until now, there is no automated support for finding and prioritizing system test cases regarding the fully integrated aPS that are suitable for finding regressions. Thus, the testing technician has to rely on personal intuition and experience, possibly choosing an inappropriate order of test cases, finding regressions at a very late stage of the test run. Using a suitable prioritization, this iterative process of finding and fixing regressions can be streamlined and a lot of time can be saved by executing test cases likely to identify new regressions earlier. Thus, an approach is presented in this paper that uses previously acquired runtime data from past test executions and performs a change identification and impact analysis to prioritize test cases that have a high probability to unveil regressions caused by side effects of a system change. The approach was developed in cooperation with reputable industrial partners active in the field of aPS engineering, ensuring a development in line with industrial requirements. An industrial case study and an expert evaluation were performed, showing promising results.