Autonomous Large Language Model Agents Enabling Intent-Driven Mobile GUI Testing

TL;DR

This paper introduces DroidAgent, an autonomous LLM-based agent for intent-driven GUI testing on Android, achieving higher coverage and realistic task execution compared to existing methods.

Contribution

It presents DroidAgent, a novel LLM-powered autonomous agent that performs high-level, intent-driven GUI testing for Android apps, surpassing current automated testing tools.

Findings

DroidAgent achieved 61% activity coverage on average.

It autonomously created 317 realistic, relevant tasks.

DroidAgent outperformed state-of-the-art techniques in coverage.

Abstract

GUI testing checks if a software system behaves as expected when users interact with its graphical interface, e.g., testing specific functionality or validating relevant use case scenarios. Currently, deciding what to test at this high level is a manual task since automated GUI testing tools target lower level adequacy metrics such as structural code coverage or activity coverage. We propose DroidAgent, an autonomous GUI testing agent for Android, for semantic, intent-driven automation of GUI testing. It is based on Large Language Models and support mechanisms such as long- and short-term memory. Given an Android app, DroidAgent sets relevant task goals and subsequently tries to achieve them by interacting with the app. Our empirical evaluation of DroidAgent using 15 apps from the Themis benchmark shows that it can set up and perform realistic tasks, with a higher level of autonomy. For example, when testing a messaging app, DroidAgent created a second account and added a first account as a friend, testing a realistic use case, without human intervention. On average, DroidAgent achieved 61% activity coverage, compared to 51% for current state-of-the-art GUI testing techniques. Further, manual analysis shows that 317 out of the 374 autonomously created tasks are realistic and relevant to app functionalities, and also that DroidAgent interacts deeply with the apps and covers more features.