SAINT: Service-level Integration Test Generation with Program Analysis and LLM-based Agents

TL;DR

SAINT is a novel white-box testing approach that combines static analysis, large language models, and agent-based workflows to automatically generate effective service-level tests for enterprise Java applications, improving coverage and fault detection.

Contribution

SAINT introduces a new method integrating static analysis and LLM-based agents for automated, scenario-based service-level test generation in enterprise Java applications.

Findings

SAINT achieves higher code and database interaction coverage.

SAINT detects faults more effectively than existing tools.

Developers endorse scenario-based tests generated by SAINT.

Abstract

Enterprise applications are typically tested at multiple levels, with service-level testing playing an important role in validating application functionality. Existing service-level testing tools, especially for RESTful APIs, often employ fuzzing and/or depend on OpenAPI specifications which are not readily available in real-world enterprise codebases. Moreover, these tools are limited in their ability to generate functional tests that effectively exercise meaningful scenarios. In this work, we present SAINT, a novel white-box testing approach for service-level testing of enterprise Java applications. SAINT combines static analysis, large language models (LLMs), and LLM-based agents to automatically generate endpoint and scenario-based tests. The approach builds two key models: an endpoint model, capturing syntactic and semantic information about service endpoints, and an operation dependency graph, capturing inter-endpoint ordering constraints. SAINT then employs LLM-based agents to generate tests. Endpoint-focused tests aim to maximize code and database interaction coverage. Scenario-based tests are synthesized by extracting application use cases from code and refining them into executable tests via planning, action, and reflection phases of the agentic loop. We evaluated SAINT on eight Java applications, including a proprietary enterprise application. Our results illustrate the effectiveness of SAINT in coverage, fault detection, and scenario generation. Moreover, a developer survey provides strong endorsement of the scenario-based tests generated by SAINT. Overall, our work shows that combining static analysis with agentic LLM workflows enables more effective, functional, and developer-aligned service-level test generation.