APITestGenie: Generating Web API Tests from Requirements and API Specifications with LLMs

TL;DR

APITestGenie uses LLMs, RAG, and prompt engineering to automatically generate API tests from requirements and specifications, significantly reducing manual effort and uncovering hidden API defects.

Contribution

The paper introduces APITestGenie, a novel tool that automates API test generation from requirements and OpenAPI specs using advanced AI techniques, demonstrating high success and defect detection.

Findings

Generated valid tests for 89% of requirements after three attempts.

Some tests uncovered previously unknown API defects.

API complexity and documentation detail influence success rates.

Abstract

Modern software systems rely heavily on Web APIs, yet creating meaningful and executable test scripts remains a largely manual, time-consuming, and error-prone task. In this paper, we present APITestGenie, a novel tool that leverages Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and prompt engineering to automatically generate API integration tests directly from business requirements and OpenAPI specifications. We evaluated APITestGenie on 10 real-world APIs, including 8 APIs comprising circa 1,000 live endpoints from an industrial partner in the automotive domain. The tool was able to generate syntactically and semantically valid test scripts for 89\% of the business requirements under test after at most three attempts. Notably, some generated tests revealed previously unknown defects in the APIs, including integration issues between endpoints. Statistical analysis identified API complexity and level of detail in business requirements as primary factors influencing success rates, with the level of detail in API documentation also affecting outcomes. Feedback from industry practitioners confirmed strong interest in adoption, substantially reducing the manual effort in writing acceptance tests, and improving the alignment between tests and business requirements.