PALM: Path-aware LLM-based Test Generation with Comprehension

TL;DR

PALM combines symbolic path enumeration with LLM-assisted test generation, enabling systematic exploration of program paths and improved understanding of coverage through an interactive visualization frontend.

Contribution

It introduces a novel system that integrates symbolic path analysis with LLMs, avoiding complex constraint solving and enhancing user comprehension of test coverage.

Findings

PALM achieves better path coverage than baseline methods.

The interactive frontend improves user understanding of test coverage.

Participants found PALM's visualization helpful for analyzing generated tests.

Abstract

Symbolic execution is a widely used technique for test generation, offering systematic exploration of program paths through constraint solving. However, it is fundamentally constrained by the capability to model the target code, including library functions, in terms of symbolic constraints and by the capability of underlying constraint solvers. As a result, many paths involving complex features remain unanalyzed or insufficiently modeled. Recent advances in large language models (LLMs) have shown promise in generating diverse and valid test inputs. Yet, LLMs lack mechanisms for systematically enumerating program paths and often fail to cover subtle corner cases. We observe that directly prompting an LLM with the full program leads to missed coverage of interesting paths. In this paper, we present PALM, a test generation system that combines symbolic path enumeration with LLM-assisted test generation. PALM statically enumerates possible paths through AST-level analysis and transforms each into an executable variant with embedded assertions that specify the target path. This avoids the need to translate path constraints into SMT formulas, by instead constructing program variants that the LLM can interpret. Importantly, PALM provides an interactive frontend that visualizes path coverage alongside generated tests, assembling tests based on the specific paths they exercise. A user study with 12 participants demonstrates that PALM's frontend helps users better understand path coverage and identify which paths are actually exercised by PALM-generated tests through verification and visualization of their path profiles.