A framework for assessing the capabilities of code generation of constraint domain-specific languages with large language models

TL;DR

This paper introduces a comprehensive framework for evaluating how well large language models generate code for constraint domain-specific languages, focusing on correctness and well-formedness, with experiments on OCL, Alloy, and Python.

Contribution

It presents a novel, systematic evaluation framework for assessing LLMs' ability to generate DSL code, including methods like code repair and multiple attempts to improve quality.

Findings

LLMs perform better on Python than on OCL and Alloy.

Smaller context window LLMs struggle with constraint code generation.

Code repair and multiple attempts improve generated code quality.

Abstract

Large language models (LLMs) can be used to support software development tasks, e.g., through code completion or code generation. However, their effectiveness drops significantly when considering less popular programming languages such as domain-specific languages (DSLs). In this paper, we propose a generic framework for evaluating the capabilities of LLMs generating DSL code from textual specifications. The generated code is assessed from the perspectives of well-formedness and correctness. This framework is applied to a particular type of DSL, constraint languages, focusing our experiments on OCL and Alloy and comparing their results to those achieved for Python, a popular general-purpose programming language. Experimental results show that, in general, LLMs have better performance for Python than for OCL and Alloy. LLMs with smaller context windows such as open-source LLMs may be unable to generate constraint-related code, as this requires managing both the constraint and the domain model where it is defined. Moreover, some improvements to the code generation process such as code repair (asking an LLM to fix incorrect code) or multiple attempts (generating several candidates for each coding task) can improve the quality of the generated code. Meanwhile, other decisions like the choice of a prompt template have less impact. All these dimensions can be systematically analyzed using our evaluation framework, making it possible to decide the most effective way to set up code generation for a particular type of task.