Scalable Similarity-Aware Test Suite Minimization with Reinforcement Learning

TL;DR

This paper introduces TripRL, a reinforcement learning-based method that efficiently minimizes large test suites by balancing coverage, fault detection, and similarity, achieving scalable and effective results.

Contribution

TripRL combines ILP and reinforcement learning with graph embeddings to improve scalability and effectiveness in large-scale test suite minimization.

Findings

Runtime scales linearly with problem size

Produces solutions within 47 minutes for large datasets

Maintains coverage and fault detection while increasing fault detection potential

Abstract

The Multi-Criteria Test Suite Minimization (MCTSM) problem aims to remove redundant test cases, guided by adequacy criteria such as code coverage or fault detection capability. However, current techniques either exhibit a high loss of fault detection ability or face scalability challenges due to the NP-hard nature of the problem, which limits their practical utility. We propose TripRL, a novel technique that integrates traditional criteria such as statement coverage and fault detection ability with test coverage similarity into an Integer Linear Program (ILP), to produce a diverse reduced test suite with high test effectiveness. TripRL leverages bipartite graph representation and its embedding for concise ILP formulation and combines ILP with effective reinforcement learning (RL) training. This combination renders large-scale test suite minimization more scalable and enhances test effectiveness. Our empirical evaluations demonstrate that TripRL's runtime scales linearly with the magnitude of the MCTSM problem. Notably, for large test suites from the Defects4j dataset where existing approaches fail to provide solutions within a reasonable time frame, our technique consistently delivers solutions in less than 47 minutes. The reduced test suites produced by TripRL also maintain the original statement coverage and fault detection ability while having a higher potential to detect unknown faults.