Large Language Model for Discrete Optimization Problems: Evaluation and Step-by-step Reasoning

TL;DR

This study evaluates the performance of large language models like Llama-3 and ChatGPT on diverse discrete optimization problems, highlighting their strengths, limitations, and the impact of techniques like Chain-of-Thought reasoning.

Contribution

It provides a comprehensive benchmark dataset and analysis of LLMs' capabilities in large-scale discrete optimization, including insights on effective prompting strategies.

Findings

Stronger models perform better on optimization tasks.

Chain-of-Thought methods are not always effective.

Disordered datasets can improve model performance on simple problems.

Abstract

This work investigated the capabilities of different models, including the Llama-3 series of models and CHATGPT, with different forms of expression in solving discrete optimization problems by testing natural language datasets. In contrast to formal datasets with a limited scope of parameters, our dataset included a variety of problem types in discrete optimization problems and featured a wide range of parameter magnitudes, including instances with large parameter sets, integrated with augmented data. It aimed to (1) provide an overview of LLMs' ability in large-scale problems, (2) offer suggestions to those who want to solve discrete optimization problems automatically, and (3) regard the performance as a benchmark for future research. These datasets included original, expanded and augmented datasets. Among these three datasets, the original and augmented ones aimed for evaluation while the expanded one may help finetune a new model. In the experiment, comparisons were made between strong and week models, CoT methods and No-CoT methods on various datasets. The result showed that stronger model performed better reasonably. Contrary to general agreement, it also showed that CoT technique was not always effective regarding the capability of models and disordered datasets improved performance of models on easy to-understand problems, even though they were sometimes with high variance, a manifestation of instability. Therefore, for those who seek to enhance the automatic resolution of discrete optimization problems, it is recommended to consult the results, including the line charts presented in the Appendix, as well as the conclusions drawn in this study for relevant suggestions.