Automated Unit Test Generation for Python

TL;DR

This paper introduces Pynguin, a framework for automated unit test generation in Python, exploring how search-based methods perform in dynamically typed languages and the impact of dynamic typing on test generation.

Contribution

The paper presents Pynguin, the first framework for automated test generation in Python, and evaluates the effectiveness of evolutionary algorithms in this context.

Findings

Evolutionary algorithms outperform random test generation in Python.

Type inference can mitigate some issues caused by dynamic typing.

Dynamic typing remains a fundamental challenge for automated test generation.

Abstract

Automated unit test generation is an established research field, and mature test generation tools exist for statically typed programming languages such as Java. It is, however, substantially more difficult to automatically generate supportive tests for dynamically typed programming languages such as Python, due to the lack of type information and the dynamic nature of the language. In this paper, we describe a foray into the problem of unit test generation for dynamically typed languages. We introduce Pynguin, an automated unit test generation framework for Python. Using Pynguin, we aim to empirically shed light on two central questions: (1) Do well-established search-based test generation methods, previously evaluated only on statically typed languages, generalise to dynamically typed languages? (2) What is the influence of incomplete type information and dynamic typing on the problem of automated test generation? Our experiments confirm that evolutionary algorithms can outperform random test generation also in the context of Python, and can even alleviate the problem of absent type information to some degree. However, our results demonstrate that dynamic typing nevertheless poses a fundamental issue for test generation, suggesting future work on integrating type inference.