Addressing the Regression Test Problem with Change Impact Analysis for Ada

TL;DR

This paper introduces a change impact analysis approach for Ada that ensures safe regression test selection, significantly reducing test execution while maintaining correctness in high-criticality software development.

Contribution

It presents the first practical change impact analysis method for Ada, combining static and dynamic data to select safe test subsets in regression testing.

Findings

97% reduction in test execution on Ada DNS server example

Effective safe test selection for high-criticality Ada projects

Validated approach on open-source Ada examples

Abstract

The regression test selection problem--selecting a subset of a test-suite given a change--has been studied widely over the past two decades. However, the problem has seen little attention when constrained to high-criticality developments and where a "safe" selection of tests need to be chosen. Further, no practical approaches have been presented for the programming language Ada. In this paper, we introduce an approach to solving the selection problem given a combination of both static and dynamic data for a program and a change-set. We present a change impact analysis for Ada that selects the safe set of tests that need to be re-executed to ensure no regressions. We have implemented the approach in the commercial, unit-testing tool VectorCAST, and validated it on a number of open-source examples. On an example of a fully-functioning Ada implementation of a DNS server (IRONSIDES), the experimental results show a 97% reduction in test-case execution.