LLM-Meta-SR: In-Context Learning for Evolving Selection Operators in Symbolic Regression

TL;DR

This paper introduces a meta-learning framework using LLMs to design effective selection operators for symbolic regression, overcoming limitations like lack of semantic guidance and code bloat, leading to state-of-the-art results.

Contribution

The paper presents a novel LLM-based meta-learning approach that incorporates semantic awareness and bloat control to improve symbolic regression algorithms.

Findings

LLMs can generate selection operators that outperform nine expert-designed baselines.

The evolved operators enhance existing symbolic regression algorithms to achieve top performance.

The approach outperforms 28 algorithms across 116 datasets, demonstrating superior effectiveness.

Abstract

Large language models (LLMs) have revolutionized algorithm development, yet their application in symbolic regression, where algorithms automatically discover symbolic expressions from data, remains limited. In this paper, we propose a meta-learning framework that enables LLMs to automatically design selection operators for evolutionary symbolic regression algorithms. We first identify two key limitations in existing LLM-based algorithm evolution techniques: lack of semantic guidance and code bloat. The absence of semantic awareness can lead to ineffective exchange of useful code components, while bloat results in unnecessarily complex components; both can hinder evolutionary learning progress or reduce the interpretability of the designed algorithm. To address these issues, we enhance the LLM-based evolution framework for meta-symbolic regression with two key innovations: a complementary, semantics-aware selection operator and bloat control. Additionally, we embed domain knowledge into the prompt, enabling the LLM to generate more effective and contextually relevant selection operators. Our experimental results on symbolic regression benchmarks show that LLMs can devise selection operators that outperform nine expert-designed baselines, achieving state-of-the-art performance. Moreover, the evolved operator can further improve a state-of-the-art symbolic regression algorithm, achieving the best performance among 28 symbolic regression and other machine learning algorithms across 116 regression datasets. This demonstrates that LLMs can exceed expert-level algorithm design for symbolic regression.