OptiHive: Ensemble Selection for LLM-Based Optimization via Statistical Modeling

TL;DR

OptiHive is a framework that improves LLM-based optimization solvers by generating diverse components, filtering errors, and using statistical models to select high-quality solvers, significantly boosting performance on complex problems.

Contribution

It introduces a novel ensemble selection method combining diverse generated components with statistical performance modeling for optimization.

Findings

Increases optimality rate from 5% to 92% on complex problems.

Effectively filters erroneous components to ensure interpretability.

Outperforms baseline methods across various optimization tasks.

Abstract

LLM-based solvers have emerged as a promising means of automating problem modeling and solving. However, they remain unreliable and often depend on iterative repair loops that result in significant latency. We introduce OptiHive, a framework that enhances any solver-generation pipeline to produce higher-quality solvers from natural-language descriptions of optimization problems. OptiHive uses a single batched generation to produce diverse components (solvers, problem instances, and validation tests) and filters out erroneous components to ensure fully interpretable outputs. Accounting for the imperfection of the generated components, we employ a statistical model to infer their true performance, enabling principled uncertainty quantification and solver selection. On tasks ranging from traditional optimization problems to challenging variants of the Multi-Depot Vehicle Routing Problem, OptiHive significantly outperforms baselines, increasing the optimality rate from 5% to 92% on the most complex problems.