Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics

TL;DR

GLMTest is a novel program structure-aware language model framework that uses code semantics and graph neural networks to generate targeted test cases, improving branch coverage and bug discovery.

Contribution

It introduces a new approach combining code property graphs with language models for controllable, branch-targeted test case generation.

Findings

GLMTest improves branch accuracy from 27.4% to 50.2%.

It outperforms Claude-Sonnet-4.5 and GPT-4o-mini on TestGenEval.

Structured conditioning enhances bug and security vulnerability detection.

Abstract

Recent advances in large language models for test case generation have improved branch coverage via prompt-engineered mutations. However, they still lack principled mechanisms for steering models toward specific high-risk execution branches, limiting their effectiveness for discovering subtle bugs and security vulnerabilities. We propose GLMTest, the first program structure-aware LLM framework for targeted test case generation that seamlessly integrates code property graphs and code semantics using a graph neural network and a language model to condition test case generation on execution branches. This structured conditioning enables controllable and branch-targeted test case generation, thereby potentially enhancing bug and security risk discovery. Experiments on real-world projects show that GLMTest built on a Qwen2.5-Coder-7B-Instruct model improves branch accuracy from 27.4% to 50.2% on TestGenEval benchmark compared with state-of-the-art LLMs, i.e., Claude-Sonnet-4.5 and GPT-4o-mini.