Teaching LLMs to Think Mathematically: A Critical Study of Decision-Making via Optimization

TL;DR

This paper critically examines how large language models can formulate and solve decision-making problems through mathematical programming, highlighting progress and limitations, and proposing future research directions.

Contribution

It provides a systematic review and empirical evaluation of LLMs' ability to generate optimization models, along with a structured roadmap for future advancements in mathematical programming with LLMs.

Findings

LLMs show promising ability to parse natural language into symbolic formulations

Significant limitations exist in accuracy, scalability, and interpretability of LLM-generated models

Future work should focus on structured datasets, domain-specific fine-tuning, and hybrid approaches

Abstract

This paper investigates the capabilities of large language models (LLMs) in formulating and solving decision-making problems using mathematical programming. We first conduct a systematic review and meta-analysis of recent literature to assess how well LLMs understand, structure, and solve optimization problems across domains. The analysis is guided by critical review questions focusing on learning approaches, dataset designs, evaluation metrics, and prompting strategies. Our systematic evidence is complemented by targeted experiments designed to evaluate the performance of state-of-the-art LLMs in automatically generating optimization models for problems in computer networks. Using a newly constructed dataset, we apply three prompting strategies: Act-as-expert, chain-of-thought, and self-consistency, and evaluate the obtained outputs based on optimality gap, token-level F1 score, and compilation accuracy. Results show promising progress in LLMs' ability to parse natural language and represent symbolic formulations, but also reveal key limitations in accuracy, scalability, and interpretability. These empirical gaps motivate several future research directions, including structured datasets, domain-specific fine-tuning, hybrid neuro-symbolic approaches, modular multi-agent architectures, and dynamic retrieval via chain-of-RAGs. This paper contributes a structured roadmap for advancing LLM capabilities in mathematical programming.