FormalSpecCpp: A Dataset of C++ Formal Specifications created using LLMs

TL;DR

FormalSpecCpp is a pioneering dataset of C++ programs with formal specifications, created using LLMs, to facilitate benchmarking and advancement in program verification and specification inference.

Contribution

It introduces the first comprehensive C++ formal specifications dataset, enabling evaluation of inference tools and LLM fine-tuning for automated specification generation.

Findings

Provides a structured benchmark for specification inference tools

Enables fine-tuning of LLMs for automated specification generation

Supports research in program verification and AI-assisted development

Abstract

FormalSpecCpp is a dataset designed to fill the gap in standardized benchmarks for verifying formal specifications in C++ programs. To the best of our knowledge, this is the first comprehensive collection of C++ programs with well-defined preconditions and postconditions. It provides a structured benchmark for evaluating specification inference tools and testing theaccuracy of generated specifications. Researchers and developers can use this dataset to benchmark specification inference tools,fine-tune Large Language Models (LLMs) for automated specification generation, and analyze the role of formal specifications in improving program verification and automated testing. By making this dataset publicly available, we aim to advance research in program verification, specification inference, and AI-assisted software development. The dataset and the code are available at https://github.com/MadhuNimmo/FormalSpecCpp.