Let's Have a Conversation: Designing and Evaluating LLM Agents for Interactive Optimization

TL;DR

This paper introduces a scalable methodology for evaluating large language model-based decision agents through conversations, demonstrating improved solution quality in interactive optimization tasks like school scheduling.

Contribution

It presents a novel conversation-based evaluation framework for optimization agents and shows how tailored, domain-specific prompts enhance their performance.

Findings

Conversation-based evaluation reveals higher-quality solutions than one-shot methods.

Tailored optimization agents outperform general-purpose chatbots in fewer interactions.

Operations research expertise improves the design and reliability of interactive optimization agents.

Abstract

Optimization is as much about modeling the right problem as solving it. Identifying the right objectives, constraints, and trade-offs demands extensive interaction between researchers and stakeholders. Large language models can empower decision-makers with optimization capabilities through interactive optimization agents that can propose, interpret and refine solutions. However, it is fundamentally harder to evaluate a conversation-based interaction than traditional one-shot approaches. This paper proposes a scalable and replicable methodology for evaluating optimization agents through conversations. We build LLM-powered decision agents that role-play diverse stakeholders, each governed by an internal utility function but communicating like a real decision-maker. We generate thousands of conversations in a school scheduling case study. Results show that one-shot evaluation is severely limiting: the same optimization agent converges to much higher-quality solutions through conversations. Then, this paper uses this methodology to demonstrate that tailored optimization agents, endowed with domain-specific prompts and structured tools, can lead to significant improvements in solution quality in fewer interactions, as compared to general-purpose chatbots. These findings provide evidence of the benefits of emerging solutions at the AI-optimization interface to expand the reach of optimization technologies in practice. They also uncover the impact of operations research expertise to facilitate interactive deployments through the design of effective and reliable optimization agents.