Assured Automatic Programming via Large Language Models

TL;DR

This paper introduces a novel method for improving AI-generated code reliability by discovering and formalizing programmer intent through a co-evolutionary repair process involving code, specifications, and tests, leading to more verifiable programs.

Contribution

It proposes a new program-proof co-evolution approach that refines programmer intent and enhances code correctness verification in AI-assisted programming.

Findings

Increased percentage of verifiable auto-generated programs.

Formalized unambiguous programmer intent.

Effective refinement of code, specification, and test.

Abstract

With the advent of AI-based coding engines, it is possible to convert natural language requirements to executable code in standard programming languages. However, AI-generated code can be unreliable, and the natural language requirements driving this code may be ambiguous. In other words, the intent may not be accurately captured in the code generated from AI-coding engines like Copilot. The goal of our work is to discover the programmer intent, while generating code which conforms to the intent and a proof of this conformance. Our approach to intent discovery is powered by a novel repair engine called program-proof co-evolution, where the object of repair is a tuple (code, logical specification, test) generated by an LLM from the same natural language description. The program and the specification capture the initial operational and declarative description of intent, while the test represents a concrete, albeit partial, understanding of the intent. Our objective is to achieve consistency between the program, the specification, and the test by incrementally refining our understanding of the user intent. Reaching consistency through this repair process provides us with a formal, logical description of the intent, which is then translated back into natural language for the developer's inspection. The resultant intent description is now unambiguous, though expressed in natural language. We demonstrate how the unambiguous intent discovered through our approach increases the percentage of verifiable auto-generated programs on a recently proposed dataset in the Dafny programming language.