DSL or Code? Evaluating the Quality of LLM-Generated Algebraic Specifications: A Case Study in Optimization at Kinaxis

TL;DR

This paper explores the effectiveness of LLMs in generating algebraic specifications, specifically AMPL models, from natural language, comparing their quality to Python code in optimization tasks through a case study with industrial data.

Contribution

It introduces EXEOS, an LLM-based method for generating and refining AMPL models and Python code from NL descriptions, demonstrating competitive performance in industrial optimization scenarios.

Findings

AMPL models are often as good as or better than Python code for optimization.

The EXEOS approach improves the quality of generated algebraic specifications.

AMLP models show promising results in real-world supply-chain cases.

Abstract

Model-driven engineering (MDE) provides abstraction and analytical rigour, but industrial adoption in many domains has been limited by the cost of developing and maintaining models. Large language models (LLMs) can help shift this cost balance by supporting direct generation of models from natural-language (NL) descriptions. For domain-specific languages (DSLs), however, LLM-generated models may be less accurate than LLM-generated code in mainstream languages such as Python, due to the latter's dominance in LLM training corpora. We investigate this issue in mathematical optimization, with AMPL, a DSL with established industrial use. We introduce EXEOS, an LLM-based approach that derives AMPL models and Python code from NL problem descriptions and iteratively refines them with solver feedback. Using a public optimization dataset and real-world supply-chain cases from our industrial partner Kinaxis, we evaluate generated AMPL models against Python code in terms of executability and correctness. An ablation study with two LLM families shows that AMPL is competitive with, and sometimes better than, Python, and that our design choices in EXEOS improve the quality of generated specifications.