BOSQTGEN: Breaking the Sound Barrier in Test Generation

TL;DR

BOSQTGEN is a black-box API test generation tool that leverages large language models and combinatorial testing to improve coverage and efficiency, addressing key challenges in modern software testing.

Contribution

It introduces a novel approach combining LLMs and combinatorial testing for structured API test generation, achieving significant coverage improvements.

Findings

Achieves 82% code coverage on RESTful benchmarks

Outperforms prior state-of-the-art systems by 20% or more

Near parity with hand-written test suites

Abstract

Modern software is increasingly built by composing APIs, elevating the API contract to a critical role. Inadequate contracts, however, lead to mismatched expectations and failures, creating a pressing need for robust conformance testing. Current test generation techniques are hindered by key challenges: polyglot systems, source code inaccessibility, a cost-reliability trade-off, and, most critically, the difficulty of generating structured inputs. We introduce BOSQTGEN, a novel black-box methodology and tool for API test generation. BOSQTGEN utilizes a novel approach for decomposing API specifications into primitives, using LLMs to suggest coherent strata for them, and employing combinatorial testing to efficiently sample over these values. This approach ensures coverage of critical interactions while avoiding the redundancy of random sampling. The resulting BOSQTGEN system achieves an average of 82% code coverage on RESTful benchmarks, often a 20% or more increase over prior state-of-the-art systems and nearing parity with hand-written test suites. Providing a fully API-driven approach to test generation, enables developers to automatically create high-quality test cases for validation or test-driven development.