SAGE:Specification-Aware Grammar Extraction for Automated Test Case Generation with LLMs

TL;DR

This paper introduces SAGE, a method that uses large language models and reinforcement learning to automatically generate valid, general grammars from natural language specifications, significantly improving test case generation quality.

Contribution

The work presents a novel approach combining LLM fine-tuning and reward-guided reinforcement learning to induce context-free grammars with counters from specifications, enhancing validity and generality.

Findings

SAGE outperforms 17 LLMs in grammar validity and test effectiveness.

The approach improves state-of-the-art by over 15% in grammar validity.

Iterative feedback enhances grammar correction for syntactic and semantic errors.

Abstract

Grammar-based test case generation has proven effective for competitive programming problems, but generating valid and general grammars from natural language specifications remains a key challenge, especially under limited supervision. Context-Free Grammars with Counters (CCFGs) have recently been introduced as a formalism to represent such specifications with logical constraints by storing and reusing counter values during derivation. In this work, we explore the use of open-source large language models (LLMs) to induce CCFGs from specifications using a small number of labeled examples and verifiable reward-guided reinforcement learning. Our approach first fine-tunes an open-source LLM to perform specification-to-grammar translation, and further applies Group Relative Policy Optimization (GRPO) to enhance grammar validity and generality. We also examine the effectiveness of iterative feedback for open and closed-source LLMs in correcting syntactic and semantic errors in generated grammars. Experimental results show that our approach SAGE achieves stronger generalization and outperforms 17 open and closed-source LLMs in both grammar quality and test effectiveness, improving over the state-of-the-art by 15.92%p in grammar validity and 12.34%p in test effectiveness. We provide our implementation and dataset at the following anonymous repository:https://anonymous.4open.science/r/SAGE-5714