LLMize: A Framework for Large Language Model-Based Numerical Optimization

TL;DR

LLMize introduces an open-source framework leveraging large language models for flexible, constraint-aware numerical optimization across diverse complex problems, emphasizing natural language interaction over traditional mathematical programming.

Contribution

The paper presents LLMize, a novel framework that enables LLM-driven optimization with iterative prompting, supporting complex constraints and domain knowledge without requiring specialized optimization expertise.

Findings

Effective on complex, domain-specific problems with constraints

Less competitive than classical solvers on simple problems

Provides accessible optimization for practitioners without formal mathematical skills

Abstract

Large language models (LLMs) have recently shown strong reasoning capabilities beyond traditional language tasks, motivating their use for numerical optimization. This paper presents LLMize, an open-source Python framework that enables LLM-driven optimization through iterative prompting and in-context learning. LLMize formulates optimization as a black-box process in which candidate solutions are generated in natural language, evaluated by an external objective function, and refined over successive iterations using solution-score feedback. The framework supports multiple optimization strategies, including Optimization by Prompting (OPRO) and hybrid LLM-based methods inspired by evolutionary algorithms and simulated annealing. A key advantage of LLMize is the ability to inject constraints, rules, and domain knowledge directly through natural language descriptions, allowing practitioners to define complex optimization problems without requiring expertise in mathematical programming or metaheuristic design. LLMize is evaluated on convex optimization, linear programming, the Traveling Salesman Problem, neural network hyperparameter tuning, and nuclear fuel lattice optimization. Results show that while LLM-based optimization is not competitive with classical solvers for simple problems, it provides a practical and accessible approach for complex, domain-specific tasks where constraints and heuristics are difficult to formalize.