MASTEST: A LLM-Based Multi-Agent System For RESTful API Tests

TL;DR

MASTEST is a multi-agent system leveraging large language models to automate the entire RESTful API testing workflow, from scenario generation to bug detection, with human-in-the-loop review for quality assurance.

Contribution

This paper introduces MASTEST, a novel multi-agent framework combining LLMs and programmed agents for comprehensive API testing, including scenario creation, script generation, execution, and analysis.

Findings

DeepSeek achieves high data type correctness and status code detection.

GPT-4o excels in API operation coverage.

Generated test scripts are 100% syntactically correct with minimal manual edits.

Abstract

Testing RESTful API is increasingly important in quality assurance of cloud-native applications. Recent advances in machine learning (ML) techniques have demonstrated that various testing activities can be performed automatically by large language models (LLMs) with reasonable accuracy. This paper develops a multi-agent system called MASTEST that combines LLM-based and programmed agents to form a complete tool chain that covers the whole workflow of API test starting from generating unit and system test scenarios from API specification in the OpenAPI Swagger format, to generating of Pytest test scripts, executing test scripts to interact with web services, to analysing web service response messages to determine test correctness and calculate test coverage. The system also supports the incorporation of human testers in reviewing and correcting LLM generated test artefacts to ensure the quality of testing activities. MASTEST system is evaluated on two LLMs, GPT-4o and DeepSeek V3.1 Reasoner with five public APIs. The performances of LLMs on various testing activities are measured by a wide range of metrics, including unit and system test scenario coverage and API operation coverage for the quality of generated test scenarios, data type correctness, status code coverage and script syntax correctness for the quality of LLM generated test scripts, as well as bug detection ability and usability of LLM generated test scenarios and scripts. Experiment results demonstrated that both DeepSeek and GPT-4o achieved a high overall performance. DeepSeek excels in data type correctness and status code detection, while GPT-4o performs best in API operation coverage. For both models, LLM generated test scripts maintained 100\% syntax correctness and only required minimal manual edits for semantic correctness. These findings indicate the effectiveness and feasibility of MASTEST.