Leveraging Large Language Models for Solving Rare MIP Challenges

TL;DR

This paper introduces a recursive dynamic temperature method combined with chain-of-thought prompting to enhance large language models' ability to solve rare and complex MIP problems more efficiently, complementing traditional solvers.

Contribution

It proposes a novel recursive temperature scheduling technique with chain-of-thought prompting to improve LLM performance on rare MIP challenges, demonstrating better solutions and efficiency.

Findings

High-to-low temperature scheduling yields better feasible solutions.

LLMs can complement traditional solvers by accelerating pruning.

Dynamic temperature strategies outperform fixed or other adaptive methods.

Abstract

Mixed Integer Programming (MIP) has been extensively applied in areas requiring mathematical solvers to address complex instances within tight time constraints. However, as the problem scale increases, the complexity of model formulation and finding feasible solutions escalates significantly. In contrast, the model-building cost for end-to-end models, such as large language models (LLMs), remains largely unaffected by problem scale due to their pattern recognition capabilities. While LLMs, like GPT-4, without fine-tuning, can handle some traditional medium-scale MIP problems, they struggle with uncommon or highly specialized MIP scenarios. Fine-tuning LLMs can yield some feasible solutions for medium-scale MIP instances, but these models typically fail to explore diverse solutions when constrained by a low and constant temperature, limiting their performance. In this paper, we propose and evaluate a recursively dynamic temperature method integrated with a chain-of-thought approach. Our findings show that starting with a high temperature and gradually lowering it leads to better feasible solutions compared to other dynamic temperature strategies. Additionally, by comparing results generated by the LLM with those from Gurobi, we demonstrate that the LLM can produce solutions that complement traditional solvers by accelerating the pruning process and improving overall efficiency.