Position: Leverage Foundational Models for Black-Box Optimization

TL;DR

This paper explores how foundational language models, especially sequence-based models like Transformers, can revolutionize black-box optimization by leveraging textual information, improving search strategies, and predicting performance in unseen spaces.

Contribution

It frames black-box optimization around sequence models and discusses innovative ways to integrate foundational models into optimization processes.

Findings

Foundational models can enhance task understanding through textual data.

Sequence models like Transformers can improve optimization strategies.

Performance prediction over unseen spaces can be significantly improved.

Abstract

Undeniably, Large Language Models (LLMs) have stirred an extraordinary wave of innovation in the machine learning research domain, resulting in substantial impact across diverse fields such as reinforcement learning, robotics, and computer vision. Their incorporation has been rapid and transformative, marking a significant paradigm shift in the field of machine learning research. However, the field of experimental design, grounded on black-box optimization, has been much less affected by such a paradigm shift, even though integrating LLMs with optimization presents a unique landscape ripe for exploration. In this position paper, we frame the field of black-box optimization around sequence-based foundation models and organize their relationship with previous literature. We discuss the most promising ways foundational language models can revolutionize optimization, which include harnessing the vast wealth of information encapsulated in free-form text to enrich task comprehension, utilizing highly flexible sequence models such as Transformers to engineer superior optimization strategies, and enhancing performance prediction over previously unseen search spaces.