Are We Testing or Being Tested? Exploring the Practical Applications of Large Language Models in Software Testing

TL;DR

This paper investigates how large language models are practically applied in industrial software testing, highlighting their benefits and limitations through a survey of professional testers and real-world data analysis.

Contribution

It provides empirical insights into the current use of LLMs in software testing, emphasizing their practical benefits and the need for cautious adoption with guidelines.

Findings

LLMs improve testing documentation and automate test case generation.

They assist testers in debugging and coding tasks.

Caution is advised for early-stage adoption without proper guidelines.

Abstract

A Large Language Model (LLM) represents a cutting-edge artificial intelligence model that generates coherent content, including grammatically precise sentences, human-like paragraphs, and syntactically accurate code snippets. LLMs can play a pivotal role in software development, including software testing. LLMs go beyond traditional roles such as requirement analysis and documentation and can support test case generation, making them valuable tools that significantly enhance testing practices within the field. Hence, we explore the practical application of LLMs in software testing within an industrial setting, focusing on their current use by professional testers. In this context, rather than relying on existing data, we conducted a cross-sectional survey and collected data within real working contexts, specifically, engaging with practitioners in industrial settings. We applied quantitative and qualitative techniques to analyze and synthesize our collected data. Our findings demonstrate that LLMs effectively enhance testing documents and significantly assist testing professionals in programming tasks like debugging and test case automation. LLMs can support individuals engaged in manual testing who need to code. However, it is crucial to emphasize that, at this early stage, software testing professionals should use LLMs with caution while well-defined methods and guidelines are being built for the secure adoption of these tools.