Exploring the True Potential: Evaluating the Black-box Optimization Capability of Large Language Models

TL;DR

This paper systematically evaluates large language models' capabilities in black-box optimization, revealing their limitations in numerical tasks but potential in non-numerical and heuristic-based problems.

Contribution

It provides the first comprehensive analysis of LLMs in numerical and non-numerical optimization, highlighting their strengths and weaknesses.

Findings

LLMs perform poorly on pure numerical tasks due to domain mismatch.

LLMs can solve non-numerical problems and leverage heuristics from prompts.

This work offers new insights into LLMs' role in diverse optimization scenarios.

Abstract

Large language models (LLMs) have demonstrated exceptional performance not only in natural language processing tasks but also in a great variety of non-linguistic domains. In diverse optimization scenarios, there is also a rising trend of applying LLMs. However, whether the application of LLMs in the black-box optimization problems is genuinely beneficial remains unexplored. This paper endeavors to offer deep insights into the potential of LLMs in optimization through a comprehensive investigation, which covers both discrete and continuous optimization problems to assess the efficacy and distinctive characteristics that LLMs bring to this field. Our findings reveal both the limitations and advantages of LLMs in optimization. Specifically, on the one hand, despite the significant power consumed for running the models, LLMs exhibit subpar performance in pure numerical tasks, primarily due to a mismatch between the problem domain and their processing capabilities; on the other hand, although LLMs may not be ideal for traditional numerical optimization, their potential in broader optimization contexts remains promising, where LLMs exhibit the ability to solve problems in non-numerical domains and can leverage heuristics from the prompt to enhance their performance. To the best of our knowledge, this work presents the first systematic evaluation of LLMs for numerical optimization. Our findings pave the way for a deeper understanding of LLMs' role in optimization and guide future application of LLMs in a wide range of scenarios.