Dynamic Bayesian Optimization Framework for Instruction Tuning in Partial Differential Equation Discovery

TL;DR

This paper introduces NeuroSymBO, a Bayesian Optimization-based framework that adaptively selects instructions for large language models to improve partial differential equation discovery, overcoming prompt brittleness and enhancing solution accuracy.

Contribution

It presents a novel adaptive instruction selection method using Bayesian Optimization, significantly improving PDE discovery performance over static prompts.

Findings

NeuroSymBO outperforms fixed prompt methods in PDE benchmarks.

Adaptive instruction selection yields higher recovery rates.

The approach produces more parsimonious solutions.

Abstract

Large Language Models (LLMs) show promise for equation discovery, yet their outputs are highly sensitive to prompt phrasing, a phenomenon we term instruction brittleness. Static prompts cannot adapt to the evolving state of a multi-step generation process, causing models to plateau at suboptimal solutions. To address this, we propose NeuroSymBO, which reframes prompt engineering as a sequential decision problem. Our method maintains a discrete library of reasoning strategies and uses Bayesian Optimization to select the optimal instruction at each step based on numerical feedback. Experiments on PDE discovery benchmarks show that adaptive instruction selection significantly outperforms fixed prompts, achieving higher recovery rates with more parsimonious solutions.