SBFT Tool Competition 2024 -- Python Test Case Generation Track
Nicolas Erni, Al-Ameen Mohammed Ali Mohammed, Christian Birchler, and Pouria Derakhshanfar, Stephan Lukasczyk, Sebastiano Panichella

TL;DR
This paper reports on the first Python Test Case Generation Competition, evaluating four tools on open-source projects to analyze their effectiveness in code and mutation coverage amidst the challenges of Python's dynamic nature.
Contribution
It introduces the organization, methodology, and results of the inaugural Python test case generation competition, highlighting current challenges and performance of existing tools.
Findings
Klara achieved the highest code coverage.
Hypothesis Ghostwriter showed strong mutation coverage.
Tools faced significant challenges due to Python's dynamic features.
Abstract
Test case generation (TCG) for Python poses distinctive challenges due to the language's dynamic nature and the absence of strict type information. Previous research has successfully explored automated unit TCG for Python, with solutions outperforming random test generation methods. Nevertheless, fundamental issues persist, hindering the practical adoption of existing test case generators. To address these challenges, we report on the organization, challenges, and results of the first edition of the Python Testing Competition. Four tools, namely UTBotPython, Klara, Hypothesis Ghostwriter, and Pynguin were executed on a benchmark set consisting of 35 Python source files sampled from 7 open-source Python projects for a time budget of 400 seconds. We considered one configuration of each tool for each test subject and evaluated the tools' effectiveness in terms of code and mutation…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsScientific Computing and Data Management · Computational Physics and Python Applications
