An approach for Test Impact Analysis on the Integration Level in Java programs

TL;DR

This paper presents a runtime analysis-based approach for selecting relevant integration tests in Java programs to reduce testing overhead, applicable in CI/CD pipelines, with significant test reduction demonstrated in open-source and commercial systems.

Contribution

It introduces a novel method for test impact analysis at the integration level using runtime analysis, addressing the challenge of linking code changes to tests across repositories.

Findings

Approximately 50% reduction in tests on average

Over 80% reduction in some cases

Effective in both open-source and commercial systems

Abstract

Test Impact Analysis is an approach to obtain a subset of tests impacted by code changes. This approach is mainly applied to unit testing where the link between the code and its associated tests is easy to obtain. On the integration level, however, it is not straightforward to find such a link programmatically, especially when the integration tests are held into separate repositories. We propose an approach for selecting integration tests based on the runtime analysis of code changes to reduce the test execution overhead. We provide a set of tools and a framework that can be plugged into existing CI/CD pipelines. We have evaluated the approach on a range of open-source Java programs and found $\approx$50\% reduction in tests on average, and above 80\% in a few cases. We have also applied the approach to a large-scale commercial system in production and found similar results.