Test-Oriented Programming: rethinking coding for the GenAI era

TL;DR

This paper introduces Test-Oriented Programming (TOP), a new paradigm where developers focus on verifying test code generated from natural language specifications, leveraging LLMs to handle production code.

Contribution

It proposes TOP as a novel approach to software development using LLMs, shifting focus from writing production code to validating automatically generated tests.

Findings

Generated test code effectively captures specifications.

Proof-of-concept tool successfully created a command-line program.

Identified challenges for applying TOP in real-world projects.

Abstract

Large language models (LLMs) have shown astonishing capability of generating software code, leading to its use to support developers in programming. Proposed tools have relied either on assistants for improved auto-complete or multi-agents, in which different model instances are orchestrated to perform parts of a problem to reach a complete solution. We argue that LLMs can enable a higher-level of abstraction, a new paradigm we called Test-Oriented Programming (TOP). Within this paradigm, developers only have to check test code generated based on natural language specifications, rather than focusing on production code, which could be delegated to the LLMs. To evaluate the feasibility of this proposal, we developed a proof-of-concept tool and used it to generate a small command-line program employing two different LLMs. We obtained promising results and identified challenges for the use of this paradigm for real projects.