Semantics for probabilistic programming: higher-order functions, continuous distributions, and soft constraints

TL;DR

This paper develops a semantic foundation for expressive probabilistic programming languages with higher-order functions, continuous distributions, and soft constraints, providing formal semantics and validating key inference algorithms.

Contribution

It introduces a metalanguage for probabilistic computation with advanced features, develops operational and denotational semantics, and proves their soundness, adequacy, and termination.

Findings

Validated correctness of compiler optimisations and inference algorithms

Enabled defining probability distributions on higher-order functions

Provided intuitive computational readings involving measure theory and stochastic systems

Abstract

We study the semantic foundation of expressive probabilistic programming languages, that support higher-order functions, continuous distributions, and soft constraints (such as Anglican, Church, and Venture). We define a metalanguage (an idealised version of Anglican) for probabilistic computation with the above features, develop both operational and denotational semantics, and prove soundness, adequacy, and termination. They involve measure theory, stochastic labelled transition systems, and functor categories, but admit intuitive computational readings, one of which views sampled random variables as dynamically allocated read-only variables. We apply our semantics to validate nontrivial equations underlying the correctness of certain compiler optimisations and inference algorithms such as sequential Monte Carlo simulation. The language enables defining probability distributions on higher-order functions, and we study their properties.