Can ChatGPT support software verification?

TL;DR

This paper explores the potential of ChatGPT to support formal software verification by generating loop invariants, demonstrating its ability to produce valid invariants that enhance verification success.

Contribution

It presents the first investigation into using ChatGPT for loop invariant generation, showing its capability to generate useful invariants that improve verification outcomes.

Findings

ChatGPT can generate valid loop invariants for C programs.

Generated invariants enable verifiers to solve previously unsolvable tasks.

The study proposes combining language models with formal verifiers for improved software verification.

Abstract

Large language models have become increasingly effective in software engineering tasks such as code generation, debugging and repair. Language models like ChatGPT can not only generate code, but also explain its inner workings and in particular its correctness. This raises the question whether we can utilize ChatGPT to support formal software verification. In this paper, we take some first steps towards answering this question. More specifically, we investigate whether ChatGPT can generate loop invariants. Loop invariant generation is a core task in software verification, and the generation of valid and useful invariants would likely help formal verifiers. To provide some first evidence on this hypothesis, we ask ChatGPT to annotate 106 C programs with loop invariants. We check validity and usefulness of the generated invariants by passing them to two verifiers, Frama-C and CPAchecker. Our evaluation shows that ChatGPT is able to produce valid and useful invariants allowing Frama-C to verify tasks that it could not solve before. Based on our initial insights, we propose ways of combining ChatGPT (or large language models in general) and software verifiers, and discuss current limitations and open issues.