An Empirical Study of Automated Unit Test Generation for Python

TL;DR

This paper presents an extended version of the Pynguin framework for automated unit test generation in Python, evaluating multiple algorithms and highlighting current challenges like type inference.

Contribution

It extends Pynguin to support more Python features, compares state-of-the-art algorithms, and evaluates regression assertion generation for Python testing.

Findings

Evolutionary algorithms outperform random test generation in Python.

DynaMOSA achieves the highest code coverage among tested algorithms.

Type inference remains a key challenge limiting test generation effectiveness.

Abstract

Various mature automated test generation tools exist for statically typed programming languages such as Java. Automatically generating unit tests for dynamically typed programming languages such as Python, however, is substantially more difficult due to the dynamic nature of these languages as well as the lack of type information. Our Pynguin framework provides automated unit test generation for Python. In this paper, we extend our previous work on Pynguin to support more aspects of the Python language, and by studying a larger variety of well-established state of the art test-generation algorithms, namely DynaMOSA, MIO, and MOSA. Furthermore, we improved our Pynguin tool to generate regression assertions, whose quality we also evaluate. Our experiments confirm that evolutionary algorithms can outperform random test generation also in the context of Python, and similar to the Java world, DynaMOSA yields the highest coverage results. However, our results also demonstrate that there are still fundamental remaining issues, such as inferring type information for code without this information, currently limiting the effectiveness of test generation for Python.