DCP-Bench-Open: Evaluating LLMs for Constraint Modelling of Discrete Combinatorial Problems

TL;DR

This paper introduces DCP-Bench-Open, a diverse benchmark dataset for evaluating large language models' ability to perform constraint modelling on a wide range of discrete combinatorial problems, addressing a key bottleneck in the field.

Contribution

It provides a new, diverse benchmark dataset for evaluating LLMs in constraint modelling, including multiple problem types and modelling frameworks, and systematically assesses various LLM and prompting strategies.

Findings

Higher performance with high-level Python frameworks

Prompt-based and inference-time methods improve accuracy

Achieved up to 91% accuracy on the benchmark

Abstract

Discrete Combinatorial Problems (DCPs) are prevalent in industrial decision-making and optimisation. However, while constraint solving technologies for DCPs have advanced significantly, the core process of formalising them, namely constraint modelling, requires significant expertise and remains a bottleneck for wider adoption. Aiming to alleviate this bottleneck, recent studies have explored using Large Language Models (LLMs) to transform combinatorial problem descriptions into executable constraint models. However, the existing evaluation datasets for discrete constraint modelling are often limited to small, homogeneous, or domain-specific problems, which do not capture the diversity of real-world scenarios. This work addresses this gap by introducing DCP-Bench-Open, a novel benchmark that includes a diverse set of well-known discrete combinatorial problems sourced from the Constraint Programming (CP) and Operations Research (OR) communities, structured explicitly for evaluating LLM-driven constraint modelling. With this dataset, and given the variety of modelling frameworks, we compare and evaluate the modelling capabilities of LLMs for three distinct constraint modelling systems, which vary in abstraction level and underlying syntax. Notably, the results show higher performance when modelling with a high-level Python-based framework. Additionally, we systematically evaluate the use of prompt-based and inference-time compute methods across different LLMs, which further increase accuracy, reaching up to 91% on this highly challenging benchmark. DCP-Bench-Open is publicly available.