Production Monitoring to Improve Test Suites

TL;DR

This paper introduces PANKTI, a Java-based approach that leverages production executions to automatically generate differential unit tests and oracles, enhancing test suite coverage and quality for real-world applications.

Contribution

PANKTI is a novel method that uses production data to improve test suites by generating targeted differential tests and oracles for Java applications.

Findings

PANKTI successfully generates differential tests from production data.

Generated tests improve existing test suite coverage.

Evaluation on three open-source projects confirms effectiveness.

Abstract

In this paper, we propose to use production executions to improve the quality of testing for certain methods of interest for developers. These methods can be methods that are not covered by the existing test suite, or methods that are poorly tested. We devise an approach called PANKTI which monitors applications as they execute in production, and then automatically generates differential unit tests, as well as derived oracles, from the collected data. PANKTI's monitoring and generation focuses on one single programming language, Java. We evaluate it on three real-world, open-source projects: a videoconferencing system, a PDF manipulation library, and an e-commerce application. We show that PANKTI is able to generate differential unit tests by monitoring target methods in production, and that the generated tests improve the quality of the test suite of the application under consideration.