A Probabilistic Framework for LLM-Based Model Discovery

TL;DR

This paper introduces a probabilistic inference framework for discovering scientific models using LLMs, enabling more systematic and interpretable model proposal, refinement, and selection processes.

Contribution

It recasts LLM-based model discovery as probabilistic inference and introduces ModelSMC, a Sequential Monte Carlo algorithm for iterative model refinement.

Findings

ModelSMC effectively discovers interpretable scientific models.

The probabilistic framework improves model proposal and refinement.

Experiments show enhanced posterior predictive accuracy.

Abstract

Automated methods for discovering mechanistic simulator models from observational data offer a promising path toward accelerating scientific progress. Such methods often take the form of agentic-style iterative workflows that repeatedly propose and revise candidate models by imitating human discovery processes. However, existing LLM-based approaches typically implement such workflows via hand-crafted heuristic procedures, without an explicit probabilistic formulation. We recast model discovery as probabilistic inference, i.e., as sampling from an unknown distribution over mechanistic models capable of explaining the data. This perspective provides a unified way to reason about model proposal, refinement, and selection within a single inference framework. As a concrete instantiation of this view, we introduce ModelSMC, an algorithm based on Sequential Monte Carlo sampling. ModelSMC represents candidate models as particles which are iteratively proposed and refined by an LLM, and weighted using likelihood-based criteria. Experiments on real-world scientific systems illustrate that this formulation discovers models with interpretable mechanisms and improves posterior predictive checks. More broadly, this perspective provides a probabilistic lens for understanding and developing LLM-based approaches to model discovery.