LICO: Large Language Models for In-Context Molecular Optimization

TL;DR

LICO leverages large language models with specialized training to perform in-context molecular optimization, enabling effective predictions on unseen properties and achieving state-of-the-art results on benchmark tasks.

Contribution

The paper introduces LICO, a novel method that adapts large language models for black-box optimization in molecular domains through in-context training and prompting.

Findings

LICO performs competitively on the PMO benchmark.

LICO achieves state-of-the-art results on PMO-1K.

The approach generalizes to unseen molecule properties.

Abstract

Optimizing black-box functions is a fundamental problem in science and engineering. To solve this problem, many approaches learn a surrogate function that estimates the underlying objective from limited historical evaluations. Large Language Models (LLMs), with their strong pattern-matching capabilities via pretraining on vast amounts of data, stand out as a potential candidate for surrogate modeling. However, directly prompting a pretrained language model to produce predictions is not feasible in many scientific domains due to the scarcity of domain-specific data in the pretraining corpora and the challenges of articulating complex problems in natural language. In this work, we introduce LICO, a general-purpose model that extends arbitrary base LLMs for black-box optimization, with a particular application to the molecular domain. To achieve this, we equip the language model with a separate embedding layer and prediction layer, and train the model to perform in-context predictions on a diverse set of functions defined over the domain. Once trained, LICO can generalize to unseen molecule properties simply via in-context prompting. LICO performs competitively on PMO, a challenging molecular optimization benchmark comprising 23 objective functions, and achieves state-of-the-art performance on its low-budget version PMO-1K.