VeriAct: Beyond Verifiability -- Agentic Synthesis of Correct and Complete Formal Specifications

TL;DR

This paper introduces VeriAct, a novel framework that uses verification-guided, agentic synthesis to generate correct and complete formal specifications, surpassing existing prompt-based methods.

Contribution

The paper presents VeriAct, an innovative agentic approach that iteratively synthesizes and repairs formal specifications using verification feedback, improving correctness and completeness.

Findings

Prompt optimization improves verifier pass rates but hits a performance ceiling.

Spec-Harness reveals many specifications accepted by verifiers are incorrect or incomplete.

VeriAct outperforms baseline methods in producing verifiable, correct, and complete specifications.

Abstract

Formal specifications play a central role in ensuring software reliability and correctness. However, automatically synthesizing high-quality formal specifications remains a challenging task, often requiring domain expertise. Recent work has applied large language models to generate specifications in Java Modeling Language (JML), reporting high verification pass rates. But does passing a verifier mean that the specification is actually correct and complete? In this work, we first conduct a comprehensive evaluation comparing classical and prompt-based approaches for automated JML specification synthesis. We then investigate whether prompt optimization can push synthesis quality further by evolving prompts through structured verification feedback. While optimization improves verifier pass rates, we find a clear performance ceiling. More critically, we propose Spec-Harness, an evaluation framework that measures specification correctness and completeness through symbolic verification, revealing that a large fraction of verifier-accepted specifications, including optimized ones, are in fact incorrect or incomplete, over- or under-constraining both inputs and outputs in ways invisible to the verifier. To push beyond this ceiling, we propose VeriAct, a verification-guided agentic framework that iteratively synthesizes and repairs specifications through a closed loop of LLM-driven planning, code execution, verification, and Spec-Harness feedback. Our experiments on two benchmark datasets show that VeriAct outperforms both prompt-based and prompt-optimized baselines, producing specifications that are not only verifiable but also correct and complete.