PyTester: Deep Reinforcement Learning for Text-to-Testcase Generation

TL;DR

PyTester leverages deep reinforcement learning to generate accurate, executable test cases directly from natural language requirements, supporting true test-driven development without needing source code as input.

Contribution

This paper introduces PyTester, a novel deep RL-based approach for text-to-testcase generation that works from requirements, outperforming larger models on the APPS benchmark.

Findings

PyTester outperforms GPT-3.5, StarCoder, and InCoder on the APPS dataset.

Deep RL enables small models to generate high-quality test cases.

The approach supports true TDD by generating test cases from requirements alone.

Abstract

Test-driven development (TDD) is a widely-employed software development practice that mandates writing test cases based on requirements before writing the actual code. While writing test cases is the centerpiece of TDD, it is time-consuming, expensive, and often shunned by developers. To address these issues associated with TDD, automated test case generation approaches have recently been investigated. Such approaches take source code as input, but not the requirements. Therefore, existing work does not fully support true TDD, as actual code is required to generate test cases. In addition, current deep learning-based test case generation approaches are trained with one learning objective, i.e., to generate test cases that are exactly matched with the ground-truth test cases. However, such approaches may limit the model's ability to generate different yet correct test cases. In this paper, we introduce PyTester, a Text-to-Testcase generation approach that can automatically generate syntactically correct, executable, complete, and effective test cases while being aligned with a given natural language requirement. We evaluate PyTester on the public APPS benchmark dataset, and the results show that our Deep RL approach enables PyTester, a small language model, to outperform much larger language models like GPT3.5, StarCoder, and InCoder. Our findings suggest that future research could consider improving small over large LMs for better resource efficiency by integrating the SE domain knowledge into the design of reinforcement learning architecture.