Can ChatGPT advance software testing intelligence? An experience report on metamorphic testing

TL;DR

This paper explores ChatGPT's potential to enhance software testing by generating metamorphic relation candidates, demonstrating its usefulness while emphasizing the need for human validation to ensure correctness.

Contribution

It presents a case study showing ChatGPT's ability to generate metamorphic relations for software testing, highlighting its role as a supportive tool requiring human oversight.

Findings

ChatGPT can generate correct metamorphic relations for some software systems.

Most MR candidates are vague or incorrect without human refinement.

ChatGPT can assist in advancing software testing but cannot replace human judgment.

Abstract

While ChatGPT is a well-known artificial intelligence chatbot being used to answer human's questions, one may want to discover its potential in advancing software testing. We examine the capability of ChatGPT in advancing the intelligence of software testing through a case study on metamorphic testing (MT), a state-of-the-art software testing technique. We ask ChatGPT to generate candidates of metamorphic relations (MRs), which are basically necessary properties of the object program and which traditionally require human intelligence to identify. These MR candidates are then evaluated in terms of correctness by domain experts. We show that ChatGPT can be used to generate new correct MRs to test several software systems. Having said that, the majority of MR candidates are either defined vaguely or incorrect, especially for systems that have never been tested with MT. ChatGPT can be used to advance software testing intelligence by proposing MR candidates that can be later adopted for implementing tests; but human intelligence should still inevitably be involved to justify and rectify their correctness.