Automated Testing of the GUI of a Real-Life Engineering Software using Large Language Models

TL;DR

This paper introduces GERALLT, an automated system leveraging Large Language Models to perform exploratory GUI testing on real-world engineering software, identifying interface issues efficiently.

Contribution

The paper presents GERALLT, a novel LLM-based system for automated GUI testing that detects interface issues in real-world engineering software.

Findings

GERALLT successfully identified interface issues in the engineering software.

The system's findings support the software development team in improving usability.

GERALLT reduces time and effort compared to manual testing methods.

Abstract

One important step in software development is testing the finished product with actual users. These tests aim, among other goals, at determining unintuitive behavior of the software as it is presented to the end-user. Moreover, they aim to determine inconsistencies in the user-facing interface. They provide valuable feedback for the development of the software, but are time-intensive to conduct. In this work, we present GERALLT, a system that uses Large Language Models (LLMs) to perform exploratory tests of the Graphical User Interface (GUI) of a real-life engineering software. GERALLT automatically generates a list of potential unintuitive and inconsistent parts of the interface. We present the architecture of GERALLT and evaluate it on a real-world use case of the engineering software, which has been extensively tested by developers and users. Our results show that GERALLT is able to determine issues with the interface that support the software development team in future development of the software.