AI4BayesCode: From Natural Language Descriptions to Validated Modular Stateful Bayesian Samplers

TL;DR

AI4BayesCode is an AI-driven system that translates natural language Bayesian descriptions into validated, modular MCMC samplers, enhancing reliability and extensibility in probabilistic programming.

Contribution

It introduces a novel, modular, stateful coding paradigm for MCMC and a validation framework, enabling automatic, reliable generation of Bayesian samplers from natural language.

Findings

Successfully implements a wide range of Bayesian models from natural language.

Demonstrates improved reliability through validation at multiple stages.

Shows extensibility with new models and components as AI improves.

Abstract

Coding and computation remain major bottlenecks in Markov chain Monte Carlo (MCMC) workflows, especially as modern sampling algorithms have become increasingly complex and existing probabilistic programming systems remain limited in model support, extensibility, and composability. We introduce \textbf{AI4BayesCode}, an extensible LLM-driven system that translates natural-language Bayesian model descriptions into runnable, validated MCMC samplers. To improve reliability, AI4BayesCode adopts a modular design that decomposes models into modular sampling blocks and maps each block to a built-in sampling component, reducing the need to implement complex sampling algorithms from scratch. Reliability is further improved through pre-generation validation of model specifications and post-generation validation of generated sampler code. AI4BayesCode also introduces a novel recursively stateful coding paradigm for MCMC, allowing modular sampling components, potentially developed by different contributors, to be composed coherently within larger MCMC procedures. We develop a benchmark suite to evaluate AI4BayesCode for sampler-generation. Experiments show that AI4BayesCode can implement a wide range of Bayesian models from natural-language descriptions alone. As an open-ended system, its capability can continue to expand with improvements in the underlying AI agent and the addition of new built-in blocks.