A Systematic Approach for Assessing Large Language Models' Test Case Generation Capability

TL;DR

This paper introduces GBCV, a systematic framework for generating diverse programs to evaluate large language models' ability to create effective test cases, revealing strengths and limitations of current models.

Contribution

The paper presents GBCV, a novel benchmark generation approach that enables comprehensive assessment of LLMs' test case creation across varied programming scenarios.

Findings

GPT-4o performs better on complex structures

Models detect boundary values in simple conditions

Challenges remain in arithmetic computation tasks

Abstract

Software testing ensures the quality and reliability of software products, but manual test case creation is labor-intensive. With the rise of large language models (LLMs), there is growing interest in unit test creation with LLMs. However, effective assessment of LLM-generated test cases is limited by the lack of standardized benchmarks that comprehensively cover diverse programming scenarios. To address the assessment of LLM's test case generation ability and lacking dataset for evaluation, we propose the Generated Benchmark from Control-Flow Structure and Variable Usage Composition (GBCV) approach, which systematically generates programs used for evaluating LLMs' test generation capabilities. By leveraging basic control-flow structures and variable usage, GBCV provides a flexible framework to create a spectrum of programs ranging from simple to complex. Because GPT-4o and GPT-3-Turbo are publicly accessible models, to present real-world regular user's use case, we use GBCV to assess LLM performance on them. Our findings indicate that GPT-4o performs better on complex program structures, while all models effectively detect boundary values in simple conditions but face challenges with arithmetic computations. This study highlights the strengths and limitations of LLMs in test generation, provides a benchmark framework, and suggests directions for future improvement.