Boundary Value Test Input Generation Using Prompt Engineering with LLMs: Fault Detection and Coverage Analysis

TL;DR

This paper evaluates the use of large language models with prompt engineering for boundary value test input generation, analyzing their fault detection and coverage capabilities compared to traditional methods.

Contribution

It introduces a framework for assessing LLMs in boundary value testing, highlighting their strengths and limitations in fault detection and coverage analysis.

Findings

LLMs can effectively detect common boundary-related issues.

They face challenges with complex or less common test inputs.

LLMs show potential but require improvements for comprehensive boundary testing.

Abstract

As software systems grow more complex, automated testing has become essential to ensuring reliability and performance. Traditional methods for boundary value test input generation can be time-consuming and may struggle to address all potential error cases effectively, especially in systems with intricate or highly variable boundaries. This paper presents a framework for assessing the effectiveness of large language models (LLMs) in generating boundary value test inputs for white-box software testing by examining their potential through prompt engineering. Specifically, we evaluate the effectiveness of LLM-based test input generation by analyzing fault detection rates and test coverage, comparing these LLM-generated test sets with those produced using traditional boundary value analysis methods. Our analysis shows the strengths and limitations of LLMs in boundary value generation, particularly in detecting common boundary-related issues. However, they still face challenges in certain areas, especially when handling complex or less common test inputs. This research provides insights into the role of LLMs in boundary value testing, underscoring both their potential and areas for improvement in automated testing methods.