BOSQTGEN: Breaking the Sound Barrier in Test Generation
S M Sadrul Islam Asif, James Chen, Earl T. Barr, Mark Marron

TL;DR
BOSQTGEN is a black-box API test generation tool that leverages large language models and combinatorial testing to improve coverage and efficiency, addressing key challenges in modern software testing.
Contribution
It introduces a novel approach combining LLMs and combinatorial testing for structured API test generation, achieving significant coverage improvements.
Findings
Achieves 82% code coverage on RESTful benchmarks
Outperforms prior state-of-the-art systems by 20% or more
Near parity with hand-written test suites
Abstract
Modern software is increasingly built by composing APIs, elevating the API contract to a critical role. Inadequate contracts, however, lead to mismatched expectations and failures, creating a pressing need for robust conformance testing. Current test generation techniques are hindered by key challenges: polyglot systems, source code inaccessibility, a cost-reliability trade-off, and, most critically, the difficulty of generating structured inputs. We introduce BOSQTGEN, a novel black-box methodology and tool for API test generation. BOSQTGEN utilizes a novel approach for decomposing API specifications into primitives, using LLMs to suggest coherent strata for them, and employing combinatorial testing to efficiently sample over these values. This approach ensures coverage of critical interactions while avoiding the redundancy of random sampling. The resulting BOSQTGEN system achieves…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSoftware Testing and Debugging Techniques · Software Engineering Research · Software System Performance and Reliability
