Toward a Trustworthy Optimization Modeling Agent via Verifiable Synthetic Data Generation

TL;DR

This paper introduces a verifiable synthetic data pipeline and a modular LLM agent, OptiTrust, for trustworthy optimization modeling, achieving state-of-the-art accuracy and full verifiability in solving linear and mixed-integer linear programming problems.

Contribution

It presents a novel framework for generating verifiable synthetic data and a multi-stage LLM agent tailored for optimization modeling tasks.

Findings

Achieves highest accuracy on six out of seven benchmark datasets.

Outperforms existing algorithms by at least 8% on three datasets.

Provides a scalable and verifiable approach for real-world optimization applications.

Abstract

We present a framework for training trustworthy large language model (LLM) agents for optimization modeling via a verifiable synthetic data generation pipeline. Focusing on linear and mixed-integer linear programming, our approach begins with structured symbolic representations and systematically produces natural language descriptions, mathematical formulations, and solver-executable code. By programmatically constructing each instance with known optimal solutions, the pipeline ensures full verifiability and enables automatic filtering of low-quality demonstrations generated by teacher models. Each dataset instance includes a structured representation of the optimization problem, a corresponding natural language description, the verified optimal solution, and step-by-step demonstrations - generated by a teacher model - that show how to model and solve the problem across multiple optimization modeling languages. This enables supervised fine-tuning of open-source LLMs specifically tailored to optimization tasks. To operationalize this pipeline, we introduce OptiTrust, a modular LLM agent that performs multi-stage translation from natural language to solver-ready code, leveraging stepwise demonstrations, multi-language inference, and majority-vote cross-validation. Our agent achieves state-of-the-art performance on standard benchmarks. Out of 7 datasets, it achieves the highest accuracy on six and outperforms the next-best algorithm by at least 8 percentage on three of them. Our approach provides a scalable, verifiable, and principled path toward building reliable LLM agents for real-world optimization applications.