Generative AI for Test Driven Development: Preliminary Results

TL;DR

This paper explores using Generative AI, specifically ChatGPT, to automate Test Driven Development, comparing collaborative and fully-automated patterns, and evaluates their impact on test quality, code quality, and development speed.

Contribution

It introduces a novel approach to automate TDD with GenAI and compares interaction patterns, providing preliminary insights into its effectiveness and challenges.

Findings

GenAI can be efficiently used in TDD with supervision.

Supervision is necessary to ensure code quality.

GenAI may mislead non-expert developers in some cases.

Abstract

Test Driven Development (TDD) is one of the major practices of Extreme Programming for which incremental testing and refactoring trigger the code development. TDD has limited adoption in the industry, as it requires more code to be developed and experienced developers. Generative AI (GenAI) may reduce the extra effort imposed by TDD. In this work, we introduce an approach to automatize TDD by embracing GenAI either in a collaborative interaction pattern in which developers create tests and supervise the AI generation during each iteration or a fully-automated pattern in which developers only supervise the AI generation at the end of the iterations. We run an exploratory experiment with ChatGPT in which the interaction patterns are compared with the non-AI TDD regarding test and code quality and development speed. Overall, we found that, for our experiment and settings, GenAI can be efficiently used in TDD, but it requires supervision of the quality of the produced code. In some cases, it can even mislead non-expert developers and propose solutions just for the sake of the query.