Combining Type Inference and Automated Unit Test Generation for Python

TL;DR

This paper introduces a method combining type inference with automated unit test generation in Python, using runtime type tracing to improve test coverage and type information accuracy.

Contribution

It presents a novel type tracing extension for Pynguin that enhances test generation and type inference in dynamically-typed Python programs.

Findings

Up to 90% increase in branch coverage

Improved mutation scores

Type information comparable to state-of-the-art tools

Abstract

Automated unit test generation is an established research field that has so far focused on statically-typed programming languages. The lack of type information in dynamically-typed programming languages, such as Python, inhibits test generators, which heavily rely on information about parameter and return types of functions to select suitable arguments when constructing test cases. Since automated test generators inherently rely on frequent execution of candidate tests, we make use of these frequent executions to address this problem by introducing type tracing, which extracts type-related information during execution and gradually refines the available type information. We implement type tracing as an extension of the Pynguin test-generation framework for Python, allowing it (i) to infer parameter types by observing how parameters are used during runtime, (ii) to record the types of values that function calls return, and (iii) to use this type information to increase code coverage. The approach leads to up to 90.0% more branch coverage, improved mutation scores, and to type information of similar quality to that produced by other state-of-the-art type-inference tools.