Beyond Code Generation: Assessing Code LLM Maturity with Postconditions

TL;DR

This paper introduces a new maturity model for code LLMs based on postcondition generation, providing a more comprehensive evaluation of their understanding and generation capabilities beyond traditional code synthesis benchmarks.

Contribution

The paper proposes a novel maturity model for code LLMs centered on postcondition generation, expanding evaluation beyond code writing to include understanding and semantic capabilities.

Findings

Open-source models need significant improvements in understanding code semantics.

The postcondition benchmark reveals gaps in current code LLM capabilities.

Augmented EvalPlus dataset enables more comprehensive assessment of code LLMs.

Abstract

Most existing code Large Language Model (LLM) benchmarks, e.g., EvalPlus, focus on the code generation tasks. Namely, they contain a natural language description of a problem and ask the LLM to write code to solve the problem. We argue that they do not capture all capabilities needed to assess the quality of a code LLM. In this paper, we propose a code LLM maturity model, based on the postcondition generation problem, to access a more complete set of code LLM capabilities. We choose the postcondition generation problem as it requires the code LLM to understand the code including semantics, natural language, and also have the capability to generate unambiguous postconditions in programming languages (i.e., the generation capablity). Moreover, postconditions have various types, requiring different levels of these capabilities, making it suitable to evaluate the maturity of the code LLM. Based on our designed maturity model, we augment the EvalPlus dataset to a postcondition testing benchmark, and evaluated several open-sourced models. Our results highlight the necessary improvements needed for better LLMs for code. Code: https://github.com/MatureModel/PostcondGen