CATTO: Just-in-time Test Case Selection and Execution

TL;DR

This paper introduces CATTO, a tool for just-in-time test case selection in Java regression testing, reducing test suite size while maintaining fault detection, with promising preliminary results on open-source projects.

Contribution

The paper presents CATTO, a novel test case selection tool for Java that integrates with IntelliJ, optimizing regression testing by reducing test suite size without sacrificing fault detection.

Findings

Significant reduction in test suite size.

Maintains fault detection capability.

Effective on open-source Java projects.

Abstract

Regression testing ensures a System Under Test (SUT) still works as expected after changes to it. The simplest approach for regression testing consists of re-running the entire test suite against the changed version of the SUT. However, this might result in a time- and resource-consuming process; \eg when dealing with large and/or complex SUTs and test suits. To work around this problem, test Case Selection (TCS) strategies can be used. Such strategies seek to build a temporary test suite comprising only those test cases that are relevant to the changes made to the SUT, so avoiding executing those test cases that do not exercise the changed parts. In this paper, we introduce CATTO (Commit Adaptive Tool for Test suite Optimization) and CATTO INTELLIJ PLUGIN. The former is a tool implementing a TCS strategy for SUTs written in Java, while the latter is a wrapper to allow developers to use \toolName directly in IntelliJ. We also conducted a preliminary evaluation of CATTO on seven open-source Java SUTs in terms of reductions in test-suite size, fault-reveling test cases, and fault-detection capability. The results are promising and suggest that CATTO can be of help to developers when performing regression testing. The video demo and the documentation of the tool is available at: \url{https://catto-tool.github.io/}