Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

TL;DR

This paper introduces RLSQM, a reinforcement learning approach that uses static quality metrics to improve the quality of automated unit tests generated by LLMs, significantly reducing test smells and increasing correctness.

Contribution

The paper presents a novel reinforcement learning framework that optimizes LLM-generated tests for quality metrics, outperforming baseline models and even GPT-4 in test quality.

Findings

LLMs generate undesirable test smells up to 37% of the time.

RLSQM improves test quality metrics by up to 23%.

Nearly 100% syntactically-correct code achieved with RLSQM.

Abstract

Software testing is a crucial but time-consuming aspect of software development, and recently, Large Language Models (LLMs) have gained popularity for automated test case generation. However, because LLMs are trained on vast amounts of open-source code, they often generate test cases that do not adhere to best practices and may even contain test smells (anti-patterns). To address this issue, we propose Reinforcement Learning from Static Quality Metrics (RLSQM), wherein we utilize Reinforcement Learning to generate high-quality unit tests based on static analysis-based quality metrics. First, we analyzed LLM-generated tests and show that LLMs frequently do generate undesirable test smells -- up to 37% of the time. Then, we implemented lightweight static analysis-based reward model and trained LLMs using this reward model to optimize for five code quality metrics. Our experimental results demonstrate that the RL-optimized Codex model consistently generated higher-quality test cases than the base LLM, improving quality metrics by up to 23%, and generated nearly 100% syntactically-correct code. RLSQM also outperformed GPT-4 on all code quality metrics, in spite of training a substantially cheaper Codex model. We provide insights into how reliably utilize RL to improve test generation quality and show that RLSQM is a significant step towards enhancing the overall efficiency and reliability of automated software testing. Our data are available at https://doi.org/10.6084/m9.figshare.25983166.