An Agent-Based Framework for the Automatic Validation of Mathematical Optimization Models

TL;DR

This paper introduces an agent-based framework for automatically validating optimization models, inspired by software testing, to ensure correctness and requirement satisfaction.

Contribution

It presents a novel multi-agent validation approach extending software testing techniques to optimization models, with demonstrated high mutation coverage.

Findings

The method achieves high mutation coverage in validation.

The approach is supported by theoretical analysis and experimental results.

It effectively detects faults in generated optimization models.

Abstract

Recently, using Large Language Models (LLMs) to generate optimization models from natural language descriptions has became increasingly popular. However, a major open question is how to validate that the generated models are correct and satisfy the requirements defined in the natural language description. In this work, we propose a novel agent-based method for automatic validation of optimization models that builds upon and extends methods from software testing to address optimization modeling . This method consists of several agents that initially generate a problem-level testing API, then generate tests utilizing this API, and, lastly, generate mutations specific to the optimization model (a well-known software testing technique assessing the fault detection power of the test suite). In this work, we detail this validation method and show, through both theory and experiments, the high quality of validation provided by this agent ensemble in terms of the well-known software testing measure called mutation coverage.