Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast

TL;DR

This paper evaluates GPT-4o's ability to generate verifiable specifications for C programs in VeriFast, revealing strengths in preserving behavior but challenges in verifiability and redundancy issues.

Contribution

It is the first study to assess large language models' effectiveness in generating ownership logic specifications for static verifiers like VeriFast.

Findings

GPT-4o preserves functional behavior in generated specifications.

Specifications often struggle to be verifiable.

Redundancies are common in verifiable specifications.

Abstract

Static verification is a powerful method for enhancing software quality, but it demands significant human labor and resources. This is particularly true of static verifiers that reason about heap manipulating programs using an ownership logic. LLMs have shown promise in a number of software engineering activities, including code generation, test generation, proof generation for theorem provers, and specification generation for static verifiers. However, prior work has not explored how well LLMs can perform specification generation for specifications based in an ownership logic, such as separation logic. To address this gap, this paper explores OpenAI's GPT-4o model's effectiveness in generating specifications on C programs that are verifiable with VeriFast, a separation logic based static verifier. Our experiment employs three different types of user inputs as well as basic and Chain-of-Thought (CoT) prompting to assess GPT's capabilities. Our results indicate that the specifications generated by GPT-4o preserve functional behavior, but struggle to be verifiable. When the specifications are verifiable they contain redundancies. Future directions are discussed to improve the performance.