Semantics of higher-order probabilistic programs with conditioning

TL;DR

This paper develops a denotational semantics for higher-order probabilistic programs using linear operators on Banach spaces, emphasizing the treatment of randomness as a resource within a monoidal framework.

Contribution

It introduces a novel semantics based on Banach space theory that captures higher-order probabilistic computations with conditioning.

Findings

Semantics based on linear operators between Banach spaces.

Framework allows fixed point definitions for higher-order programs.

Treats randomness as a resource within a monoidal structure.

Abstract

We present a denotational semantics for higher-order probabilistic programs in terms of linear operators between Banach spaces. Our semantics is rooted in the classical theory of Banach spaces and their tensor products, but bears similarities with the well-known Scott semantics of higher-order programs through the use ordered Banach spaces which allow definitions in terms of fixed points. Being based on a monoidal rather than cartesian closed structure, our semantics effectively treats randomness as a resource.

Equations

$$\int |f{|}^{p}d\mu <\infty .$$

$$\Vert f{\Vert }_{\infty }=\inf \{C\ge 0\mid |f(x)|\le C\mu \text{-a.e.}\}$$

$$f_{\ast }(\mu )(B)={\int }_{X}f(x)(B)d\mu$$

$$\gamma (A\times B)={\int }_{A}f(x)(B)d\mu ,$$

$$f_{\mu }^{†}=({\pi }_X{)}_{\ast }\circ ({\pi }_Y{)}_{\gamma }^{†}$$

$$u\le v\Rightarrow u+w\le v+w\text{and}u\le v\Rightarrow \lambda u\le \lambda v$$

$$\Vert f{\Vert }_r=\inf \left\{\Vert g\Vert :-g\le f\le g\right\}$$

$$(\mu \vee \nu )(A)=\sup \{\mu (B)+\nu (A\setminus B)\mid B\text{ measurable },B\subseteq A\}$$

$$\Vert u+v\Vert =\Vert u\Vert +\Vert v\Vert$$

$$V_v=\{w\in V\mid (|w|\wedge n|v|)\uparrow |w|\}$$

$$-u_{\alpha }\le v_{\alpha }-v\le u_{\alpha }\text{ for all }\alpha \in D$$

$$V^{\ast }=\{f:V\to \mathbb{R}:f\text{ is norm-continuous}\}$$

$$V^{\sigma }=\{f:V\to \mathbb{R}:f\text{ is σ-order continuous and regular}\}$$

$${\mathrm{L}}_p(X,\mu {)}^{\sigma }={\mathrm{L}}_q(X,\mu )$$

$$\mathbf{RoBan}$$

$$\hat{f}:U\to W^V\text{and}\tilde{f}:V\to W^U$$

$$f(\lambda u,v)=\lambda f(u,v)\text{and}f(u,\lambda v)=\lambda f(u,v)=\lambda (u,v)$$

$$\lambda f(u,v)=f(\lambda (u,v))=f(\lambda u,\lambda v)=\lambda f(u,\lambda v)={\lambda }^{2}f(u,v)$$

$$U\times V\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}\text{ignorespaces}$$

$$(u+u^{\prime },v)=(u,v)+(u^{\prime },v),(u,v+v^{\prime })=(u,v)+(u,v^{\prime }),$$

$$(\lambda u,v)=(u,\lambda v)=\lambda (u,v)$$

$$\Vert x{\Vert }_{\pi }=\inf \left\{\sum _{i=1}^n\Vert u_i\Vert \Vert v_i\Vert :x=\sum _{i=1}^n{u}_i\otimes v_i\right\}$$

$$\Vert x{\Vert }_{\pi }=\inf \left\{\sum _{i=1}^{\infty }\Vert u_i\Vert \Vert v_i\Vert :\sum _{i=1}^{\infty }\Vert u_i\Vert \Vert v_i\Vert <\infty ,x=\sum _{i=1}^{\infty }{u}_i\otimes v_i\right\}$$

$$\Vert x{\Vert }_{|\pi |}=\inf \{\sum _{i=1}^n\Vert u_i\Vert \Vert v_i\Vert :u_i\in U^{+},v_i\in V^{+},$$

$$-\sum _{i=1}^n{u}_i\otimes v_i\le x\le \sum _{i=1}^n{u}_i\otimes v_i\}$$

$$\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{$\triangle$}\cr\scalebox{1.1}{$\otimes$}\cr}}}_{|\pi|}\circ~{}(f\times g):U\times V\to X\times Y\to X\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{$\triangle$}\cr\scalebox{1.1}{$\otimes$}\cr}}}_{|\pi|}Y$$

$$-\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{$\triangle$}\cr\scalebox{1.1}{$\otimes$}\cr}}}_{|\pi|}U\dashv[U,-]$$

$$\mathtt{T}::=$$

$$(\mathtt{T},\mu )\mid \mathtt{T}\otimes \mathtt{T}\mid \mathtt{T}\to \mathtt{T}\mid \mathtt{MT}$$

$$\mathtt{S}$$

$$\mathtt{T}$$

$$e::=$$

Full text

I Introduction

Probabilistic programming has enjoyed a recent resurgence of interest driven by new applications in machine learning and statistical analysis of large datasets. The emergence of probabilistic programming languages such as Church and Anglican, which allow statisticians to construct and sample distributions and perform Bayesian inference, has created a need for sound semantic foundations and tools for specification and reasoning. Several recent works have approached this task from various perspectives [1, 2, 3, 4].

One of the earliest works on the semantics of probabilistic programs was [5], in which operational and denotational semantics were given for an idealized first-order imperative language with random number generation. Programs and data were interpreted over ordered Banach spaces. Programs were modelled as positive and continuous linear operators on an ordered Banach space of measures. In [6], an equivalent predicate-transformer semantics was introduced based on ordered Banach spaces of measurable functions and shown to be dual to the measure-transformer semantics of [5].

In this paper revisit this approach. We identify a symmetric monoidal closed category $\mathbf{RoBan}$ of ordered Banach spaces and regular maps that can serve as a foundation for higher-order probabilistic programming with sampling, conditioning, and Bayesian inference. Bayesian inference can be viewed as reversing the computation of a probabilistic program to infer information about a prior distribution from observations. We model Bayesian inference as computing the adjoint of a linear operator and show how it corresponds to computing a disintegration.

The extension to higher types is achieved through a tensor product construction in the category $\mathbf{RoBan}$ that gives symmetric monoidal closure. Although not cartesian, the construction does admit an adjunction with homsets enriched with an ordered Banach space structure acting as internalized exponentials. To accommodate conditioning and Bayesian inference, we introduce ‘Bayesian types’, in which values are decorated with a prior distribution. Based on this foundation, we give a type system and denotational semantics for an idealized higher-order probabilistic language with sampling, conditioning, and Bayesian inference.

We believe our approach should appeal to computer scientists, as it is true to traditional Scott-style denotational semantics (see § IV-D). It should also appeal to mathematicians, statisticians and machine learning theorists, as it uses very familiar mathematical objects from those fields. For example, a traditional perspective is that a Markov process is just a positive linear operator of norm 1 between certain Banach lattices [7, Ch. 19]. These are precisely the morphisms of our semantics. Similarly, classical ergodic theory, which is key to proving the correctness of important algorithms like Gibbs sampling, is an important part of the theory of these operators [8]. Our semantics therefore connects seamlessly with a wealth of results from functional analysis, ergodic theory, statistics, etc. We believe that this will greatly simplify the task of validating stochastic machine learning algorithms.

We should also mention that our semantics fits well with the view of entropy (randomness) as a computation resource, like time or space. True random number generators can only produce randomness at a limited rate; physically, randomness is a resource [9]. Our type system, being resource-sensitive, has some nice crypographical properties: it is forbidden by construction to use a sample more than once; that is, each operation consuming a random sample requires a fresh sample (component in a tensor product).

Related works: Two very powerful semantics for higher-order probabilistic programming have been recently developed in the literature. In [1, 2], a semantics is given in terms of so-called quasi-Borel spaces. These form a Cartesian closed category and admit a notion of probability distribution and of a Giry-like monad of probability distributions. In [4] the authors develop a semantics in terms of measurable cones. These form a cpo-enriched Cartesian closed category which provides a semantics to a probabilistic extension of PCF that includes conditioning. The key differences with the present semantics are the following. First, these proposed mathematical universes come directly from the world of theoretical computer science, whilst as mentioned above, our semantics is rooted in the traditional mathematics of the objects being constructed by the programs. Second, quasi-Borel spaces and measurable cones form Cartesian closed categories, whereas we work in a monoidal closed category, with obvious implications in terms of resources (e.g. we cannot copy a value). Finally, our semantics of conditioning has been reduced to a mathematically very simple, but also very general construction (taking the adjoint of a linear operator, see § IV-B8), whilst in [1] un-normalized posteriors and normalization constants are computed pointwise, and [4] effectively hard-codes the rejection-sampling algorithm into the semantics.

The reader will find the proofs of most results in the Appendix, together with some background material on measure theory and tensor products of Banach spaces.

II Background

We start by describing the mathematical landscape of our semantics. We assume that the reader is familiar with the basic definitions of measure theory and of what a (real) Banach space is (see [7, Ch. 4, 6, 8-11] for a gentle introduction in the spirit of this paper).

II-A Banach spaces, Disintegration and Bayesian inversion

II-A1 Some important Banach spaces

Two classes of Banach spaces will appear repeatedly in this paper.

First, for any measurable space $(X,\mathcal{F})$ we introduce the space $\mathcal{M}(X,\mathcal{F})$, or simply $\mathcal{M}X$, as the set of signed measures of bounded variation over $X$. $\mathcal{M}X$ is a Banach space: the linear structure is inherited pointwise from $\mathbb{R}$, and the norm is given by the total variation; see [7, Th. 10.56] for a proof that the space is complete.

Second, for a measured space $(X,\mathcal{F},\mu)$ and $1\leq p<\infty$, the Lebesgue space $\mathrm{L}_{p}(X,\mu)$ is the set of equivalence classes of $\mu$-almost everywhere equal $p$-integrable real-valued functions, that is to say functions $f:X\to\mathbb{R}$ such that

[TABLE]

The linear structure is inherited pointwise from $\mathbb{R}$ and the norm is given by $\|f\|_{p}=\int|f|^{p}~{}d\mu$. When $p=\infty$, the space $\mathrm{L}_{\infty}(X,\mu)$ is defined as the set of equivalence classes of $\mu$-almost everywhere equal bounded real-valued functions with the norm given by the essential supremum:

[TABLE]

A proof that Lebesgue spaces are complete can be found in [7, Th. 13.5].

II-A2 Disintegration

Measurable spaces and maps form the category $\mathbf{Meas}$. We define the functor $\mathscr{M}:\mathbf{Meas}\to\mathbf{Meas}$ by setting $\mathscr{M}X$ to be the set of signed measures of bounded variation on $X$ equipped with the smallest $\sigma$-algebra making all evaluation maps $ev_{B}:\mathscr{M}X\to\mathbb{R}$, $\mu\mapsto ev_{B}(\mu)=\mu(B)$ measurable and by setting $\mathscr{M}f:\mathscr{M}X\to\mathscr{M}Y,\mu\mapsto f_{*}(\mu)$, the pushforward measure of $\mu$, for any $f:X\to Y$ 111This is just a generalisation of the Giry monad on $\mathbf{Meas}$ [10]. Note that $\mathcal{M}X$ and $\mathscr{M}X$ share the same underlying set, but the former is a Banach space and the latter a measurable space.. We define a measure kernel to be a measurable map $f:X\to\mathscr{M}Y$ such that for all $x\in X$ $\|f(x)\|<K$ for some fixed $K\in\mathbb{R}^{+}$. A probability kernel is a measure kernel such that $\|f(x)\|=1$ and $f(x)\geq 0$ for all $x\in X$. A measure $\mu\in\mathscr{M}X$ can also be pushed-forward through a measure kernel $f:X\to\mathscr{M}Y$ to give a measure in $\mathscr{M}Y$ via the definition

[TABLE]

which converges since $f(x)(B)\leq f(x)(X)\leq|f(x)|(X)=\|f(x)\|<K$.

With these definitions in place we can introduce the important notion of disintegration which underlies the semantics of Bayesian conditioning (see § IV-B8). We provide a slightly simplified version of the definition which will be enough for our purpose (see [11, Def. 1] for a very general definition). Intuitively, given a measurable map $f:X\to Y$ and a finite measure $\mu$ on $X$, we say that $\mu$ has a disintegration w.r.t. $f$ if the fibres $f^{-1}(y)$ of $f$ can be equipped with measures $f^{\dagger}_{\mu}(y)$ which average out to $\mu$ over the pushforward measure $f_{*}(\mu)$. Formally, the disintegration of $\mu$ w.r.t. to $f$ is a measure kernel $f^{\dagger}_{\mu}:Y\to\mathscr{M}X$ such that

• •

$f_{*}(f^{\dagger}_{\mu}(y))\propto\delta_{y}\text{ for }f_{*}(\mu)\text{-almost all }y\in Y$

• •

$(f^{\dagger}_{\mu})_{*}(f_{*}(\mu))=\mu$

In fact [11, Th. 3] shows that $f^{\dagger}_{\mu}$ can be chosen to be a probability kernel. As can be seen from the first condition, a disintegration – if it exists at all – is only defined up to a null set for the pushforward measure. For sufficiently well-behaved spaces, for example standard Borel spaces [12, 17.35] or more generally metric spaces with Radon measures [11, Th. 1], disintegrations can be shown to always exist.

II-A3 Bayesian inversion

The notion of disintegration is key to the understanding of Bayesian conditioning. The traditional setup is as follows: we are given a probability kernel $f:X\to\mathscr{M}Y$ where $X$ is regarded as a parameter space and $f$ is regarded as a parametrized statistical model on $Y$, a space of observable values. We also start with a probability distribution $\mu$ on $X$ (the prior) which is regarded as the current state of belief of where the ‘true’ parameters of the model lie. The problem is, given an observation $y\in Y$, to update the state of belief to a new distribution (the posterior) reflecting the observation. We must therefore find a kernel going in the opposite direction $f^{\dagger}_{\mu}:Y\to\mathscr{M}X$. As shown in [13, 14] this reverse kernel can be built using a disintegration as follows. First we define a joint distribution $\gamma\in\mathscr{M}(X\times Y)$ defined by

[TABLE]

The Bayesian inverse $f^{\dagger}$, if it exists, is given by the probability kernel

[TABLE]

where $(\pi_{Y})^{\dagger}_{\gamma}$ is the disintegration of the measure $\gamma$ along the projection $\pi_{Y}:X\times Y\to Y$ (it can be assumed to be a probability kernel). This construction clearly generalizes to all measure kernels.

II-B Ordered Banach spaces

II-B1 Regular Ordered Banach spaces

An ordered vector space $V$ is a vector space together with a partial order $\leq$ which is compatible with the linear structure in the sense that for all $u,v,w\in V,\lambda\in\mathbb{R}^{+}$

[TABLE]

A vector $v$ in an ordered vector space $V$ is called positive if $v\geq 0$ and the collection of all positive vectors is called the positive cone of $V$ and denoted $V^{+}$. The positive cone is said to be generating if $V=V^{+}-V^{+}$, that is to say if every vector can be expressed as the difference of two positive vectors.

An ordered normed vector space $V$ is an ordered vector space in which the positive cone $V^{+}$ is closed for the topology generated by the norm. A subset of the positive cone of particular importance will be the positive unit ball $B^{+}(V)=\{v\geq 0:\|v\|\leq 1\}$. An ordered Banach space is an ordered normed vector space which is complete. We can now describe the central class of object of this work: an ordered normed space is said to be regular if it satisfies [15, Ch. 9]:

1. R1

if $-y\leq x\leq y$ then $\|x\|\leq\|y\|$ 2. R2

$\|x\|=\inf\{\|y\|:-y\leq x\leq y\}$

In particular, a regular ordered Banach space is an ordered Banach space which is regular. A few comments are in order. First note that if $-y\leq y$ then $0\leq 2y$, and thus $y$ is positive, so R2 says that the norm of any vector can be approximated arbitrarily well by the norm of positive vectors. Note also that R2 implies that the positive cone is generating: for any $x\in V$, fix $\epsilon>0$, then by R2 there exists $y$ with $-y\leq x\leq y$ whose norm is $\epsilon$-close to that of $x$. Since $x=\frac{y+x}{2}-\frac{y-x}{2}$, and since it follows from $-y\leq x\leq y$ that both $y+x$ and $y-x$ are positive, $x$ can indeed be expressed as the difference of two positive vectors. Regularity can be understood as the fact that the space is fully characterised by its positive unit ball [16].

II-B2 Regular operators and $\mathbf{RoBan}$

As mentioned above, regular ordered Banach spaces are determined in a very strong sense by their positive cone which is generating and determines the norm (axiom R2). It is therefore natural to consider linear operators $f:U\to V$ between regular ordered Banach spaces which send positive vectors to positive vectors, i.e. such that $u\geq 0\Rightarrow f(u)\geq 0$. Such operators are called positive operators and constitute a field of mathematical research in their own right [17, 18]. The collection $[U,V]_{p}$ of positive operators between two regular ordered Banach spaces clearly does not form a vector space, since it is not closed under scalar multiplication by negative reals. We therefore consider the span of this collection, that is to say the operators $f:U\to V$ which can be expressed as the difference between two positive operators, i.e. $f=h-g$ with $h,g\in[U,V]_{p}$. Such operators are called regular operators, and we define the category $\mathbf{RoBan}$ as the category whose objects are regular ordered Banach spaces and whose morphisms are regular operators. Regular operators have the following important properties.

Proposition 1**.**

Regular operators on regular ordered Banach spaces are (norm) bounded.

Theorem 2** ([16]).**

If $U,V$ are regular ordered Banach spaces and $[U,V]$ is equipped with the obvious linear structure, pointwise order and the regular norm

[TABLE]

where $\|g\|=\sup\{\|g(u)\|:\|u\|\leq 1\}$ is the usual operator norm, then $[U,V]$ is a regular ordered Banach space.

This result justifies the following notation: we will denote the regular ordered Banach space of operators between the regular ordered Banach spaces $U,V$ by $[U,V]$.

II-B3 Banach lattices

We now describe a particularly important class of regular ordered Banach spaces: the class of Banach lattices. Although this class of objects lacks the categorical closure properties that we seek (see § II-D), most of the objects we will be dealing with are Banach lattices.

An ordered vector space $(V,\leq)$ is a Riesz space if its partial order is a lattice. This allows the definition of the positive and negative part of a vector $v\in V$ as $v^{+}=v\vee 0,v^{-}=(-v)\vee 0$ and its modulus as $|v|=v\vee(-v)$. Note that $v=v^{+}-v^{-}$, with $v^{+},v^{-}$ positive, and the positive cone of a Riesz space is thus generating. A Riesz space is order complete or Dedekind-complete (resp. $\sigma$-order complete or $\sigma$-Dedekind complete) if every non-empty (resp. non-empty countable) subset of $V$ which is order bounded has a supremum222Order-completeness was called conditional completeness in [5]. A normed Riesz space is a Riesz space equipped with a lattice norm, i.e. norm satisfying axiom R1 above. A normed Riesz space is called a Banach lattice if it is (norm-) complete. As stated, Banach lattices form a special class of regular ordered Banach spaces:

Proposition 3**.**

Banach lattices are regular.

Example 4**.**

Given a measurable space $(X,\mathcal{F})$, the space $\mathcal{M}(X,\mathcal{F})$ can be shown [7, Th 10.56] to be a Banach lattice. The Banach space structure was described above and the lattice structure is given by

[TABLE]

and dually for meets. The Hahn-Jordan decomposition theorem defines the positive and negative part of a measure in the Banach lattice $\mathcal{M}(X,\mathcal{F})$.

Example 5**.**

Given a measured space $(X,\mathcal{F},\mu)$ and $1\leq p\leq\infty$, the Lebesgue space $\mathrm{L}_{p}(X,\mu)$ is a Banach lattice with the pointwise order. In particular, for any $f\in\mathrm{L}_{p}(X,\mu)$, the positive and negative parts $f^{+}$ and $f^{-}$ of a function used in the definition of the Lebesgue integral defines the positive-negative decomposition of $f$ in the Banach lattice $\mathrm{L}_{p}(X,\mu)$. We will say that $p,q\in\mathbb{N}\cup\{\infty\}$ are Hölder conjugate if either of the following conditions hold: (i) $1<p,q<\infty$ and $\frac{1}{p}+\frac{1}{q}=1$, or (ii) $p=1$ and $q=\infty$, or (iii) $p=\infty$ and $q=1$.

The examples of Banach lattices described above are instances of an even better behaved class of objects called abstract Lebesgue spaces or AL spaces. They are defined by the following property of the norm: a Banach lattices $V$ is an AL space if for all $u,v\in V^{+}$

[TABLE]

Not surprisingly, the Lebesgue spaces $\mathrm{L}_{1}(X,\mu)$ are examples of AL spaces, as are the Banach lattices $\mathcal{M}(X)$.

Theorem 6** ([18], Sec. 4.1).**

AL spaces are order-complete.

II-B4 Bands

The order structure of Riesz spaces gives rise to classes of subspaces which are far richer than the traditional linear subspaces. An ideal of a Riesz space $V$ is a linear subspace $U\subseteq V$ with the property that if $|u|\leq|v|$ and $v\in U$ then $u\in U$. An ideal $U$ is called a band when for every subset $D\subseteq U$ if $\bigvee D$ exists in $V$, then it also belongs to $U$. Every band in a Banach lattice is itself a Banach lattice. Of particular importance in what follows will be the principal band generated by an element $v\in V$, which we denote $V_{v}$ and can be described explicitly by

[TABLE]

Example 7**.**

Given a measure $\mu\in\mathcal{M}X$, the band $(\mathcal{M}X)_{\mu}$ generated by $\mu$ is the set of signed measures of bounded variation which are absolutely continuous w.r.t. $\mu$ [7, Th. 10.61]. The ordered version of the Radon-Nikodym theorem states that $(\mathcal{M}X)_{\mu}\simeq\mathrm{L}_{1}(X,\mu)$ as Banach lattices [7, Th. 13.19].

II-B5 Köthe duals

There are two modes of ‘convergence’ in an ordered Banach space: order convergence and norm convergence. The latter is well-known, the former less so. Let $D$ be a directed set, and let $\{v_{\alpha}\}_{\alpha\in D}$ be a net in an ordered Banach space $V$. We say that $\{v_{\alpha}\}$ converges in order to $v$ if there exists a decreasing net $\{u_{\alpha}\}_{\alpha\in D}$ with $\bigwedge u_{\alpha}=0$, notation $u_{\alpha}\downarrow 0$, such that

[TABLE]

If the directed set $D$ is $\mathbb{N}$ we get the notion of order-convergent sequence. Order and norm convergence of sequences are disjoint concepts, i.e. neither implies the other (see [17, Ex. 15.2] for two counter-examples). However if a sequence converges both in order and in norm then the limits are the same (see [17, Th. 15.4]). Moreover, for monotone sequences norm convergence implies order convergence [17, Th. 15.3].

It is well known that bounded operators are continuous, i.e. preserve norm-converging sequences. The corresponding order-convergence concept is defined as follows: an operator $T:V\to W$ between Riesz spaces is said to be $\sigma$-order continuous if whenever $v_{n}\downarrow 0$, $Tv_{n}\downarrow 0$333Equivalently if $v_{n}\uparrow v$, i.e. $v_{n}$ is an increasing sequence with supremum $v$, implies $Tv_{n}\uparrow Tv$. Note the similarity with Scott-continuity, the only difference being the condition that sequences must be order-bounded.. We can thus consider two types of dual spaces on an ordered Banach space $V$: on the one hand we can consider the norm-dual:

[TABLE]

and the $\sigma$-order-dual:

[TABLE]

The latter is also known as the Köthe dual of $V$ [19, 17].

Theorem 8**.**

The Köthe dual $V^{\sigma}$ of a regular ordered Banach space $V$ is an order-complete Banach lattice.

Example 9**.**

It is shown in e.g. [17, 20] that

[TABLE]

for any Hölder conjugate pair $1\leq p,q\leq\infty$. In particular the spaces $\mathrm{L}_{1}(X,\mu)$ and $\mathrm{L}_{\infty}(X,\mu)$ are Köthe dual of each other. Note that they are not ordinary duals.

II-B6 Categorical connections.

We conclude this section by a summary of some results from [14] which provide a categorical connection between most of the topics covered so far.

The category $\mathbf{Krn}$ is the category whose objects are pairs $(X,\mu)$ where $X$ is standard Borel spaces [12] (in fact any class of measurable spaces for which disintegrations exist will do) and $\mu\in\mathscr{M}X$. A morphism between $(X,\mu)$ and $(Y,\nu)$ is a measure kernel $f:X\to\mathscr{M}Y$ such that $f_{*}(\mu)=\nu$ (where $f_{*}$ is defined in (1)), in which case the morphism is denoted $f$ as well. As was shown in [14], any two morphisms which disagree only on a null set can be identified, and the morphisms of $\mathbf{Krn}$ thus become equivalence classes of almost everywhere equal measure kernels (see [14] for the technical details of this construction).

Now, we define some functors. First, as was shown in [14], the Bayesian inversion operation described in § II-A3 defines a functor $(-)^{\dagger}:\mathbf{Krn}\to\mathbf{Krn}^{\mathrm{op}}$ which leaves objects unchanged and sends a morphism $f:(X,\mu)\to(Y,\nu)$ to its Bayesian inverse $f^{\dagger}:(Y,\nu)\to(X,\mu)$ (we drop the subscript $\mu$ of $f^{\dagger}_{\mu}$ because it is made explicit from the typing). Note that $(f^{\dagger})^{\dagger}=f$ [14]. We also define the functor $(-)^{\sigma}:\mathbf{RoBan}\to\mathbf{RoBan}^{\mathrm{op}}$ which sends a regular ordered Banach space $X$ to its Köthe dual, and a regular operator $T:U\to V$ to its adjoint $T^{\sigma}:V^{\sigma}\to U^{\sigma}$ defined in the usual way. Note that just as taking the Köthe dual gives an order-complete space, the adjoint $T^{\sigma}$ of a regular operator is an order-continuous regular operator [17, Ch. 26].

Connecting the categories, we define for each $1\leq p\leq\infty$ the functor $\mathrm{L}_{p}:\mathbf{Krn}\to\mathbf{RoBan}^{\mathrm{op}}$ which sends a $\mathbf{Krn}$-object $(X,\mu)$ to the Lebesgue space $\mathrm{L}_{p}(X,\mu)$ and a $\mathbf{Krn}$-arrow $f:(X,\mu)\to(Y,\nu)$ to the operator $\mathrm{L}_{p}f:\mathrm{L}_{p}(Y,\nu)\to\mathrm{L}_{p}(X,\mu)$, $\phi\mapsto\lambda x~{}.~{}\int_{Y}\phi~{}df(x)$. We also define the functor $\mathcal{M}_{-}:\mathbf{Krn}\to\mathbf{RoBan}$ which sends an object $(X,\mu)$ to the band $(\mathcal{M}X)_{\mu}$ and a morphism $f:(X,\mu)\to(Y,\nu)$ to the operator $\mathcal{M}f:(\mathcal{M}X)_{\mu}\to(\mathcal{M}Y)_{\nu}$, $\rho\mapsto\lambda B~{}.~{}\int_{X}f(x)(B)~{}d\rho$.

The functors $\mathcal{M}_{-},\mathrm{L}_{1}\circ(-)^{\dagger}$ and $(-)^{\sigma}\circ\mathrm{L}_{\infty}$ of type $\mathbf{Krn}\to\mathbf{RoBan}$ are related via natural transformations which play a major role in measure theory [14]:

• •

$\mathrm{RN}:\mathcal{M}_{-}\to\mathrm{L}_{1}\circ(-)^{\dagger}$ acts at $(X,\mu)$ by sending a measure $\nu\ll\mu$ to its Radon-Nikodym derivative $\frac{d\mu}{d\nu}$.

• •

$\mathrm{MR}:\mathrm{L}_{1}\circ(-)^{\dagger}\to\mathcal{M}_{-}$ acts at $(X,\mu)$ by sending an $\mathrm{L}_{1}$-map $f$ to its Measure Representation $f\mu$.

• •

$\mathrm{FR}:\mathcal{M}_{-}\to(-)^{\sigma}\circ\mathrm{L}_{\infty}$ acts at $(X,\mu)$ by sending a measure $\mu$ to its Functional Representation $\lambda\phi.\int\phi~{}d\mu$.

• •

$\mathrm{RR}:(-)^{\sigma}\circ\mathrm{L}_{\infty}\to\mathcal{M}_{-}$ acts at $(X,\mu)$ by sending an $\mathrm{L}_{\infty}$-functional $F$ to its Riesz Representation $\lambda B.F(1_{B})$.

The natural transformations $\mathrm{RN}$ and $\mathrm{MR}$ are inverse of each other, as are $\mathrm{FR}$ and $\mathrm{RR}$, proving natural isomorphisms between the three functors. We summarize these relationships in the following diagram:

[TABLE]

II-C Tensor products of ordered Banach spaces

We start by describing the tensor product of vector spaces from the perspective of computer science. We will then discuss how the tensor product can be normed and ordered.

II-C1 Introduction to the tensor product

As was already highlighted in [5] in the case of probabilistic programming, and subsequently in the development of semantics for quantum programming languages (e.g. [21]), it may be desirable to interpret programs as linear operators in a category of vector spaces. Indeed, this is precisely what this paper advocates for probabilistic programming languages. However, a difficulty quickly emerges if one wants to include higher-order features. Consider a map in two arguments $f:U\times V\to W$. The most basic facility provided by higher-order reasoning is the ability to curry such a map and define the two curried maps

[TABLE]

by fixing one argument or the other. Since we want both curried map $\hat{f}$ and $\tilde{f}$ to be linear, it is easy to see that $f$ must be linear in each arguments separately, in particular

[TABLE]

Such a map is referred to as a bilinear map, it is linear in each argument separately. However being bilinear is incompatible with being linear: by definition of the product linear structure if $f$ were also linear we would have

[TABLE]

which is clearly a contradiction if $\lambda\neq 1$. Thus $f$ is not a valid morphism if we want our semantic universe to consist of linear maps between vector spaces. Fortunately, for any pair of vector spaces $U,V$ there exists a special object, the tensor product $U\otimes V$, which linearizes bilinear maps, i.e. such that any bilinear map $f:U\times V\to W$ corresponds to unique linear map $\bar{f}:U\otimes V\to W$ (and vice-versa). This can be phrased in terms of a universal property: there exists a universal bilinear map $\otimes:U\times V\to U\otimes V$, $(u,v)\mapsto u\otimes v$ such that for any bilinear map $f:U\times V\to W$ there exists a unique linear map $\bar{f}:U\otimes V\to W$ making the following diagram commute:

[TABLE]

The tensor product can be built explicitly as follows: it is the free vector space over $U\times V$ quotiented by the following identities:

[TABLE]

It is not too hard to see that the last identity is precisely what is needed to fix the contradiction $\lambda f(u,v)=\lambda^{2}f(u,v)$ described above and reconcile currying with linearity. By definition, an element $x\in U\otimes V$ will be a (finite) linear combination of equivalence classes of the generators $(u,v)$ – denoted $u\otimes v$ – under the identities (5), formally $x=\sum_{i}u_{i}\otimes v_{i}$.

II-C2 Tensor product of Banach spaces

Suppose now that both $U$ and $V$ are Banach spaces, in particular that they carry a norm. How do we define a norm on $U\otimes V$, and how do we ensure that the space is complete for this norm? For Hilbert spaces there is a straightforward construction, but since we shall be dealing with Banach spaces which are not Hilbert spaces, we will require the much more subtle theory of tensor products of Banach spaces originally developed by Grothendieck [22]. We refer to [23] for a good introduction.

The initial difficulty with the construction of a norm is that by definition of the tensor product, each vector has many representations. For example, the vector $(2,2)\otimes(3,3)\in\mathbb{R}^{2}\otimes\mathbb{R}^{2}$, i.e. the equivalence class of the pair $((2,2),(3,3))$ under the equations (5), can also be expressed as $(1,1)\otimes(6,6)$, and these representations are built from vectors with very different norms. Assuming that we want the norm of the tensor be be defined from the norms of its components, which representation do we choose? There is no unique solution to this question, but Grothendieck proposed one extremal solution by defining for any $x\in U\otimes V$

[TABLE]

This definition of $\|\cdot\|_{\pi}$ defines a norm [23, Prop 2.1] which is called the projective norm. However, the space $U\otimes V$ equipped with the projective norm is in general not complete. One therefore defines the projective tensor product of two Banach spaces $U,V$ as the completion of $U\otimes V$ under the projective norm, i.e. the space of equivalence classes of Cauchy sequences in $U\otimes V$ converging to the same point. This space will be denoted U~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptV and one can describe the projective norm of elements in this space as follows:

[TABLE]

In § II-C1 we saw how the tensor product can be used as a way to linearize bilinear maps. This property extends naturally to the normed case, and it can be shown that the projective tensor product U~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptV linearizes bounded bilinear maps [23, Th 2.9] in the sense that there exists a universal bounded bilinear maps U\times V\to U~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptV satisfying the universal property of (4) w.r.t bounded bilinear maps.

The projective tensor product of two Banach spaces is in general fairly inscrutable. However, one can explicitly describe projective tensor products involving important objects for our semantics. When one component is an $\mathrm{L}_{1}$-space we have:

Theorem 10** (Radon-Nikodym and [23] p. 43).**

For finite measures $\mu,\nu$ on measurable spaces $X,Y$ respectively, (\mathcal{M}X)_{\mu}~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt(\mathcal{M}Y)_{\nu}\simeq(\mathcal{M}(X\times Y))_{\mu\times\nu}.

The operator $\times:\mathcal{M}X\times\mathcal{M}Y\to\mathcal{M}(X\times Y)$ taking the product of measures is bilinear. Therefore there exists a unique map \mathcal{M}X~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\mathcal{M}Y\to\mathcal{M}(X\times Y) mapping any tensor \mu~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\nu to the product measure. In this sense, \mathcal{M}{X}~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\mathcal{M}{Y} is the subspace of $\mathcal{M}{(X\times Y)}$ which is generated by taking linear combinations of product measures, and then closing under the projective norm.

Theorem 11**.**

The projective tensor product \mathcal{M}{X}~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\mathcal{M}{Y} is isometrically embedded in $\mathcal{M}{(X\times Y)}$.

II-C3 Tensor product of ordered Banach spaces

We conclude this brief description of tensor products by examining the case of interest to us, namely regular ordered Banach spaces. In the most important examples the construction is isomorphic as Banach spaces to the unordered case, and we will therefore not dwell too long on the theory of tensor product of ordered Banach spaces developed in [24, 25, 26]. The main idea of the definition is to reflect the central role of positive vectors in the theory of regular ordered Banach spaces, and in particular the fact that the positive cone is generating and determines the norm (axiom R1, R2 above). The same should hold for any ordered tensor product.

Given two ordered regular Banach spaces $U,V$, their tensor product $U\otimes V$ is equipped with the positive projective norm $\|\cdot\|_{|\pi|}$ defined as

[TABLE]

Note the similarity with (6), and the role played by positive vectors in this definition. As in the unordered case $U\otimes V$ is not complete for the positive projective norm, and we must therefore take its completion which we call the positive projective tensor of $U$ and $V$ and denote by U\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}V.

Since $\mathrm{L}_{1}$-spaces and $\mathcal{M}(X,\mathcal{F})$-spaces are examples of AL-spaces, the following result shows that we can in practice often ignore the subtleties of the positive projective tensor product and rely on the descriptions of the ordinary projective tensor products.

Theorem 12** ([25], Th. 2B).**

If $E$ is an $AL$-space and $F$ is any regular ordered space, then E\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}F and E~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptF are isomorphic as Banach spaces.

II-D The closed monoidal structue of $\mathbf{RoBan}$

II-D1 Tensor product of regular operators

In § II-C1 and § II-C2 we saw how the tensor and projective tensor products can be used to linearize bilinear and bounded bilinear maps respectively. The positive projective tensor product fulfils the same role for positive (and thus bounded by Prop. 1) bilinear maps: there exists a universal positive bilinear map U\times V\to U\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}V satisfying the universal property of (4) w.r.t positive bilinear maps [26, 2.7]. This universal property of tensor products provides a definition of \mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|} as a bifunctor on $\mathbf{RoBan}$. Let $f:U\to X,g:V\to Y$ be positive operators, then the map

[TABLE]

is positive and bilinear, and thus there exists a unique positive operator U\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}V\to X\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}Y which is denoted f\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}g. This provides the definition of the bifunctor \mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|} on morphisms.

II-D2 The closed monoidal structure

As we saw in Th. 2, the category $\mathbf{RoBan}$ has internal homs, and these interact correctly with positive projective tensor products.

Theorem 13** ([16]).**

For every regular ordered Banach space $U$, the tensoring and homming operations -\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}U and $[U,-]$ define functors $\mathbf{RoBan}\to\mathbf{RoBan}$ such that

[TABLE]

The positive projective tensor defines a symmetric monoidal structure on $\mathbf{RoBan}$ with $\mathbb{R}$ as its unit – since $U\otimes\mathbb{R}\simeq U\simeq\mathbb{R}\otimes U$ at the level of the underlying vector spaces – and the obvious isomorphisms U\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}V\to V\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}U inherited from the isomorphism $U\otimes V\to V\otimes U$ between the tensor product of the underlying vector spaces. The category $\mathbf{RoBan}$ is thus symmetric monoidal closed.

III A higher-order language with conditioning

III-A A type system

We start by defining a type system for our language. Our aims are to (a) have enough types to write some realistic programs for example including multivariate normal or chi-squared distributions, (b) have higher-order types, (c) provide special types for Bayesian learning: Bayesian types.

Our type grammar is given as follows:

[TABLE]

where $1\leq m,n\in\mathbb{N}$ and $\mu:\mathtt{T}$. We will refer to $m$, $\mathtt{int}^{n}$, $\mathtt{real}^{n},$ $\mathtt{PosDef}(n)$ as ground types. As their name suggest they are to be regarded as the types of (possibly random) elements of finite sets, vectors of integers, vectors of reals and positive semi-definite matrices, that is to say covariance matrices. We will write $\mathtt{int}^{1}$ and $\mathtt{real}^{1}$ as $\mathtt{int}$ and $\mathtt{real}$. This is by no means an exhaustive set of ground types, but sufficiently rich to consider some realistic probabilistic programs. The type $1\in\mathbb{N}$ will be referred to as the unit type and denoted $\mathtt{unit}$ and the type $2\in\mathbb{N}$ will be referred to as the boolean type and denoted $\mathtt{bool}$.

The type constructors are the following. First, given a term $\mu$ of type $\mathtt{T}$, we can build the pointed type $(\mathtt{T},\mu)$. We will call these types Bayesian types because $\mu:\mathtt{T}$ will be interpreted as a prior. Bayesian types will support conditioning and thus Bayesian learning. As we shall see, our Bayesian types also fulfil a role in the semantics of variable assignment. As is the tacit practise in Anglican, we will consider that assigning a (possibly random) value to a variable is equivalent to assigning a prior to the type of this variable. For example the program $\mathtt{x:=2.5}$ which assigns the value $2.5$ to the variable $\mathtt{x}$ can be understood as placing a (deterministic) prior on the reals, namely $\delta_{2.5}$. Similarly, the program $\mathtt{x:=sample(normal(0,1))}$ which assigns to $\mathtt{x}$ a value randomly sampled from the normal distribution $\mathcal{N}(0,1)$ with mean 0 and standard deviation 1 can be understood as setting the prior $\mathcal{N}(0,1)$ on the reals. In a slogan:

Bayesian types = Assigned types

Note however that this slogan is only valid for assignments without free variables, indeed a prior cannot be parametric in some variables, it represents definite information. This caveat will be reflected in the type system.

We then have two binary type constructors: tensor types and functions types which together will support higher-order reasoning. Finally, we have a unary type constructor used to define higher-order probabilities.

We isolate the following two sub-grammars of types whose semantic properties will be essential to the typing of certain operations. First we define order-complete types as the types generated by the grammar

[TABLE]

Second, we define measure types as the types generated by all the constructors of grammar (7) except the function-type constructor.

Remark 14**.**

We could easily add product types to our type system since the category in which we interpret types ($\mathbf{RoBan}$) is complete, but we feel that this would distract from the central role played by the tensor product. It would also introduce the possibility of copying, which from the perspective of randomness as a resource is problematic. This is why we have ‘hard-coded’ the products which we do need (tuples of integers and reals) as ground types.

Subtyping relation

We will need to formalise the fact that a Bayesian type $(\mathtt{T},\mu)$ is a subtype of the type $\mathtt{T}$. For this we introduce a subtyping relation denoted $<:$ freely generated by the rules

[TABLE]

As will become clear when we define the semantics of types, we can also use the subtyping relation to add information about the absolute continuity of one built-in measure w.r.t another in the type system. For example, since a beta distribution is absolutely continuous w.r.t. to a normal distribution, if $e_{1}$ is a program constructing a beta distribution and $e_{2}$ is a program constructing a normal distribution, we could add the rule

$(\mathtt{real},e_{1})<:(\mathtt{real},e_{2})$

.

Contexts

are maps $\mathtt{\Gamma}:\mathbb{N}\to\mathtt{Types}$ – the free algebra of all types generated by (7) – which send cofinitely many integers to the unit type $\mathtt{unit}$. We will write $\mathrm{supp}(\Gamma)$ for the set $\{i\mid\Gamma(i)\neq\mathtt{unit}\}$ and use the traditional notation $\mathtt{\Gamma}[i\mathtt{\mapsto\mathtt{T}}]$ to denote the context mapping $i$ to $\mathtt{T}$ and all $j\neq i$ to $\mathtt{\Gamma}(j)$. To each context we can associate the finite tensor type $\bigotimes_{i=1}^{n}\mathtt{\Gamma}(i)$ where $n=\sup\mathrm{supp}(\mathtt{\Gamma})$. When a context $\mathtt{\Gamma}$ appears to the right of a turnstile in the typing rules which follow we will implicitly perform this conversion from context to type.

Our contexts are a dynamic version of the static context of [5] which consists of a constant map on $\mathbb{N}$ to a single type. They are in some respects similar to the heap models of separation logic, and for notational clarity we will require similar operations on contexts as on heaps: a notion of compatibility, of union and of difference. Given two contexts $\mathtt{\Gamma}_{1},\mathtt{\Gamma}_{2}$ we will say that they are compatible if $\mathtt{\Gamma}_{1}(i)=\mathtt{\Gamma}_{2}(i)$ for all $i\in\mathrm{supp}(\mathtt{\Gamma}_{1})\cap\mathrm{supp}(\mathtt{\Gamma}_{2})$, and we will then write $\mathtt{\Gamma}_{i}\pitchfork\mathtt{\Gamma}_{2}$. For any two compatible contexts $\mathtt{\Gamma}_{1}\pitchfork\mathtt{\Gamma}_{2}$ we define the union context $\mathtt{\Gamma}_{1}\oplus\mathtt{\Gamma}_{2}$ as the union of their graphs, which is a function by the compatibility assumption. We define the difference context $\mathtt{\Gamma}_{1}\ominus\mathtt{\Gamma}_{2}$ as the map sending $i\mapsto\mathtt{\Gamma}_{1}(i)$ if $i\notin\mathrm{supp}(\mathtt{\Gamma}_{2})$ and to $\mathtt{unit}$ otherwise. In particular $\mathtt{\Gamma}_{1}\ominus\mathtt{\Gamma}_{2}=\mathtt{\Gamma}_{1}$ if the supports are disjoint.

III-B Syntax

We define an ML-like language allowing imperative features like variable assignments, conditionals and while loops within a functional language.

III-B1 Expressions

[TABLE]

Every built-in operation must come equipped with typing instruction which we will write as an $n+1$-tuple $(\mathtt{S_{1}},\ldots,\mathtt{S_{n}},\mathtt{T})$ where the first $n$ components are ground types specifying the input types and the last component is a ground type or measures over a ground type specifying the output type. For example the boolean connective or would come with typing $(\mathtt{bool},\mathtt{bool},\mathtt{bool})$, the sine function sin with typing $(\mathtt{real},\mathtt{real})$ and the function $\mathtt{normal}$ constructing a normal distribution would come with typing $(\mathtt{real},\mathtt{PosDef}(1),\mathtt{M}~{}\mathtt{real})$, where the first input is the mean, the second is the standard deviation (a 1-dimensional covariance matrix, i.e. a positive real) and the output is a measure over the reals.

III-B2 Well-typed expressions

The typing rules for our language are gathered in Fig. 1. We will discuss these rules in detail when we define the denotational semantics of our language in § IV, but we can already make some observations.

It is important to realize that memory-manipulating rules in effect have a sequent on the right of the turnstile, formally represented by an integer-indexed tensor product type (see § III-A), whilst the other rules just have a type. Syntactically and semantically however, we make no distinction between these two cases. A useful way to think about our system is as follows: a purely functional computation $\mathtt{\Gamma}\vdash e:\mathtt{T}$ will consume a context $\mathtt{\Gamma}$ and output a value of type $\mathtt{T}$. A program with imperative features $\mathtt{\Gamma}\vdash e:\mathtt{\Delta}$ modifying an internal store $\mathtt{\Gamma}$ to a new store $\mathtt{\Delta}$ should be thought of as consuming a context $\mathtt{\Gamma}$ and outputting the totality of its internal state $\mathtt{\Delta}$.

The only way to explicitly create a Bayesian type is through a variable assignment without free variables: a prior must contain definite information, not information which is parametric in variables. Only ‘measure types’ can form Bayesian types.

The sequential composition rule looks daunting, but it is simply a version of the cut rule with a bit of bookkeeping to make sure contexts do not conflict with one another.

Note finally that our observe statement applies to a term of type $\mathtt{T}$, intuitively we observe a possibly random element of type $\mathtt{T}$. This is slightly different from the syntax of observe in Anglican where a distribution is observed. Semantically, the difference disappears since a possibly random element is modelled by a distribution.

III-B3 A simple example

It is not hard (but notationally cumbersome) to type-check the following simple Gaussian inference program against the inference rules of Fig. 1.

⬇

let x=sample(normal(0,1)) in

observe(sample(normal(x,1)))

In the empty context the program above evaluates to a function of type

[TABLE]

which, as we will see in § IV, is what we want semantically.

IV Denotational semantics

As the reader will have guessed we will now provide a denotational semantics for the language described in § III in the category $\mathbf{RoBan}$ of regular ordered Banach spaces.

IV-A Semantics of types

For ground types we define

• •

$\llbracket m\rrbracket=\mathcal{M}\{1,\ldots,m\}$ where $\{1,\ldots,m\}$ is equipped with the discrete $\sigma$-algebra. Note that $\llbracket m\rrbracket\simeq\mathbb{R}^{m}$, and thus $\llbracket\mathtt{unit}\rrbracket\simeq\mathbb{R}$, the unit of the positive projective tensor.

• •

$\llbracket\mathtt{int}\rrbracket=\mathcal{M}\mathbb{N}$, where $\mathbb{N}$ is equipped with the discrete $\sigma$-algebra

• •

$\llbracket\mathtt{real}\rrbracket=\mathcal{M}\mathbb{R}$, where $\mathbb{R}$ is equipped with its usual Borel $\sigma$-algebra

• •

$\llbracket\mathtt{PosDef}(n)\rrbracket=\mathcal{M}\mathrm{PosDef}(n)$, where $\mathrm{PosDef}(n)$ is the space of positive semi-definite $n\times n$ matrices equipped with the Borel $\sigma$-algebra inherited from $\mathbb{R}^{n\times n}$

As expected, the tensor and function type constructors are interpreted by the monoidal closed structure of $\mathbf{RoBan}$, i.e.

[TABLE]

The higher-order probability type constructor $\mathtt{M}$ is interpreted as follows. For any regular ordered Banach space $V$ we consider the underlying set together with the Borel $\sigma$-algebra induced by the norm. We then apply the functor $\mathcal{M}$ to this measurable space. This construction is functorial and we overload $\mathcal{M}$ to denote the resulting regular ordered Banach space by $\mathcal{M}V$. Using this convenient notation we define

[TABLE]

For Bayesian types note that the type system in Fig. 1 can only produce a Bayesian type $(\mathtt{T},\mu)$ if $\mathtt{T}$ is a measure type and $\mu$ has no free variables, i.e. if $\mathtt{\emptyset}\vdash\mu:\mathtt{T}$ is derivable. We will therefore only need to provide a semantics to Bayesian types of this shape. Our semantics of Bayesian types is in some respect similar to that of pointed types used in homotopy type theory [27]. Indeed, at the type-theoretic level they are defined identically as a type together with a term inhabiting this type. However, the ordered vector space structure allows us to provide a semantics which is much richer than a space with a distinguished point. Given a measure type $\mathtt{T}$ and a sequent of the type $\mathtt{\emptyset}\vdash\mu:\mathtt{T}$, we will see in § IV-B that $\mu$ is interpreted as an operator $\llbracket\mu\rrbracket:\mathbb{R}\to\llbracket\mathtt{T}\rrbracket$, which is uniquely determined by $\llbracket\mu\rrbracket(1)$. For notational clarity we will often simply write $\mu$ for the measure $\llbracket\mu\rrbracket(1)$. We define the denotation of the Bayesian type $(\mathtt{T},\mu)$ as the principal band in $\llbracket\mathtt{T}\rrbracket$ (see § II-B4) generated by the measure $\mu$ (i.e.$\llbracket\mu\rrbracket(1)$). Formally:

[TABLE]

For this semantics to be well-defined it is necessary that $\llbracket\mathtt{T}\rrbracket$ be a Riesz space, since bands are defined using the lattice structure. This is indeed the case:

Theorem 15**.**

The semantics of a measure type is a Banach lattice.

The function type constructor is the only operation in the type system which forces us to leave the category of Banach lattices and enter the much larger category $\mathbf{RoBan}$. As shown in [18, Ex. 1.17], the space of regular operators between two Riesz spaces need not even be a lattice. The non-closure of Banach lattices under taking internal homs is one of the technical reasons for our use of ‘measure types’ 444Note that we could in principle extend (11) to all types by considering subsets generated by a single element which exist in all regular ordered Banach spaces, for example the closure of ideals..

We introduced order-complete types in § III-A because of a ‘dual’ non-closure property: order-complete spaces are not closed under the positive projective tensor operation. As shown in [25, 4C] the product \mathrm{L}_{2}(\left[0,1\right])\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\mathrm{L}_{2}(\left[0,1\right]) is not order-complete, even though $\mathrm{L}_{2}(\left[0,1\right])$ is. Order-completeness will be important in the semantics of while loops.

Theorem 16**.**

The semantics of an order-complete type is an order-complete space.

Subtypes and contexts

The subtyping relation will simply be interpreted as subspace inclusion. For example the relation $(\mathtt{T},\mu)<:\mathtt{T}$ is interpreted as the inclusion of the principal band $\llbracket\mathtt{T}\rrbracket_{\llbracket\mu\rrbracket}\hookrightarrow\llbracket\mathtt{T}\rrbracket$. A context $\mathtt{\Gamma}$ will be interpreted as the positive projective tensor

[TABLE]

and we put $\llbracket\emptyset\rrbracket:=\mathbb{R}$. A typing rule $\mathtt{\Gamma}\vdash e:\mathtt{T}$ will be interpreted as a regular (in fact positive, see Th. 23) operator $\llbracket e\rrbracket:\llbracket\mathtt{\Gamma}\rrbracket\to\llbracket\mathtt{T}\rrbracket$ .

IV-B Semantics of well-formed expressions

Let us now turn to the semantics of terms.

IV-B1 Constants

A constant $c\in G$ whose ground type $\mathtt{G}$ is interpreted as the space $\mathcal{M}G$ will be interpreted as the operator

[TABLE]

IV-B2 Built-in operations

Recall that every built-in operation $\mathtt{op}$ comes with typing information $(\mathtt{G}_{1},\ldots,\mathtt{G}_{n},\mathtt{T})$ where each $\mathtt{G}_{i}$ is of ground type and $\mathtt{T}$ is either of ground type or of type $\mathtt{M}\mathtt{G}$. Each such operation is interpreted via a function $f_{\mathtt{op}}:G_{1}\times\ldots\times G_{n}\to X$, with $X=G$ or $X=\mathcal{M}G$, as the unique regular operator which linearizes $\mathcal{M}f_{\mathtt{op}}\circ\times$ according to the universal property (4) of \mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}:

[TABLE]

For example the boolean operator or of type $(\mathtt{bool},\mathtt{bool},\mathtt{bool})$ would be interpreted, via the function $f_{\mathtt{or}}:2\times 2\to 2$ implementing the boolean join, as the linearisation of $\mathcal{M}f_{\mathtt{or}}\circ\times$ (which is bilinear). Similarly, the operation $\mathtt{normal}$ of type $(\mathtt{real},\mathtt{PosDef}(1),\mathtt{M}\mathtt{real})$ building a normal distributions would be interpreted, via the obvious function $f_{\mathtt{normal}}:\mathbb{R}\times\mathbb{R}^{+}\to\mathcal{M}\mathbb{R}$, as the linearisation of $\mathcal{M}f_{\mathtt{normal}}\circ\times$. Note that if the inputs are deterministic, i.e. a tensor $\delta_{\mu}\otimes\delta_{\sigma}$ for a mean $\mu$ and a standard deviation $\sigma$ (as would usually be the case), then $\llbracket\mathtt{normal}\rrbracket(\delta_{\mu}\otimes\delta_{\sigma})$ outputs a Dirac delta over the distribution $\mathcal{N}(\mu,\sigma)$. Note how we interpret the deterministic construction of a distribution over $X$ differently from sampling an element of $X$ according to this distribution: the former is a distribution over distributions, the latter just a distribution.

IV-B3 Variables and assignments

A variable on its own acts like a variable declaration and introduces a context (see Fig. 1). Its semantics is simply given by the identity operator on the type of the variable, formally if $[i\mapsto\mathtt{T}]\vdash x_{i}:\mathtt{T}$ then $\llbracket x_{i}\rrbracket=\mathrm{Id}_{\llbracket\mathtt{T}\rrbracket}$. In order to define the semantics of variable assignment we need the following result.555As a consequence of this theorem, the semantics of all our types are Archimedean ordered vector spaces.

Theorem 17**.**

The denotation of any type $\mathtt{T}$ admits a strictly positive functional $\phi_{\llbracket\mathtt{T}\rrbracket}$.

The strictly positive functional $\phi_{\llbracket\mathtt{T}\rrbracket}$ can be thought of as a generalisation to all types of the functional on measures which consists in evaluating the mass of the whole space, i.e. of $ev_{X}:\mathcal{M}X\to\mathbb{R},\mu\mapsto\mu(X)$. With this notion in place we can provide a semantics to assignments. Given a sequent $\mathtt{\Gamma}\vdash e:\mathtt{T}$, let $\phi_{\llbracket\mathtt{T}\rrbracket}$ be the strictly positive functional on $\llbracket\mathtt{T}\rrbracket$ constructed in Th. 17 and let us write $\llbracket\mathtt{\Gamma}[i\mapsto\mathtt{T}]\rrbracket$ as \llbracket\mathtt{\Gamma}_{1}\rrbracket\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\llbracket\mathtt{T}\rrbracket\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\llbracket\mathtt{\Gamma}_{2}\rrbracket. We now define the multilinear map

[TABLE]

This defines the unique linearizing operator666In fact a nuclear operator [28].

[TABLE]

In the case where the context $\mathtt{\Gamma}$ is empty, the premise of the typing rule for assignments is interpreted as an operator $\llbracket e\rrbracket:\mathbb{R}\to\llbracket\mathtt{T}\rrbracket$ and we can therefore strengthen the definition of $\llbracket x_{i}:=e\rrbracket$ as follows:

[TABLE]

In the empty context, the general rule for variable assignment is a consequence of the rule creating Bayesian types since $(\mathtt{T},e)<:\mathtt{T}$, i.e. there is no disagreement between the two rules.

As a simple example, it is easy to type-check the program x:=3.5 and see by unravelling the definition that it is interpreted as the operator $\mathcal{M}\mathbb{R}\to(\mathcal{M}\mathbb{R})_{\delta_{3.5}},\mu\mapsto\mu(\mathbb{R})\delta_{3.5}$. In particular any probability distribution gets mapped to $\delta_{3.5}$. This is the semantics of assignment of [5].

IV-B4 Sequencing and sequential composition

These are conceptually straightforward as they essentially implement some form of function composition. The only difficulty resides in the bookkeeping of contexts which is a bit cumbersome.

Given $\llbracket e_{1}\rrbracket:\llbracket\mathtt{\Gamma}\rrbracket\to\llbracket\mathtt{S}\rrbracket$ and $\llbracket e_{2}\rrbracket:\llbracket\mathtt{\Delta}[i\mapsto\mathtt{S}]\rrbracket\to\llbracket\mathtt{T}\rrbracket$ we define $N=\sup(\mathrm{supp}(\mathtt{\Gamma})\cup\mathrm{supp}(\mathtt{\Delta}))$, we assume w.l.o.g. that $i\notin\mathrm{supp}(\mathtt{\Gamma})$ and define the semantics of $\mathtt{let}~{}x_{i}=e_{1}~{}\mathtt{in}~{}e_{2}$ as the unique operator which linearizes the multilinear map

[TABLE]

The disjointness condition on the contexts $\mathtt{\Gamma}$ and $\mathtt{\Delta}$ implements the resource-awareness of the system: $e_{1}$ consumes the context $\mathtt{\Gamma}$, so no part of it can be re-used in the computation $e_{2}$.

Sequential composition is just a generalisation of sequencing where $e_{1}$ outputs a store and the ‘cut’ can take place over more than one type. As a simple example consider the program $\mathtt{x_{1}:=3.5~{};~{}x_{2}:=7.3}$. Using the sequential composition rule we can derive the following typing-checking proof

$\emptyset\vdash 3.5:\mathtt{real}$

$[1\mapsto\mathtt{real}]\vdash\mathtt{x_{1}:=3.5}:(\mathtt{real},3.5)$

     

$\emptyset\vdash 7.3:\mathtt{real}$

$[2\mapsto\mathtt{real}]\vdash\mathtt{x_{2}:=7.3}:(\mathtt{real},7.3)$

$[1\mapsto\mathtt{real},2\mapsto\mathtt{real}]\vdash\mathtt{x_{1}:=3.5~{};~{}x_{2}:=7.3}:(\mathtt{real},3.5)\otimes(\mathtt{real},7.3)$

The semantics of the program is the operator defined by

[TABLE]

IV-B5 $\lambda$-abstraction and function application

These are interpreted exactly as expected in a monoidal closed category, namely via the adjunction -\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\llbracket\mathtt{S}\rrbracket\dashv[\llbracket\mathtt{S}\rrbracket,-] and ordinary function application.

Remark 18**.**

While the denotation of $\lambda$-abstraction is immediately given by the monoidal closed structure of $\mathbf{RoBan}$, the following point is worth making. Assume a context of ground types only. In the system of [5], such a context is of the shape $\mathcal{M}(X_{1}\times\ldots\times X_{n})$, i.e. any joint distribution over the variables can be considered as an input to the program. If we were Cartesian closed, a context would be of the shape $\mathcal{M}X_{1}\times\ldots\times\mathcal{M}X_{n}$, i.e. only product distributions would be considered as potential inputs to the program. Our semantics lies somewhere in between these two possibilities since a context is of the shape \mathcal{M}X_{1}\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\ldots\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\mathcal{M}X_{n}. This means that not all joint probabilities can be $\lambda$-abstracted on, only those which live in the tensor product (i.e. limits of Cauchy sequences of linear combinations of product measures). Put differently, we can only $\lambda$-abstract if the probabilistic state of the machine is prepared (to use a quantum analogy) to a distribution in the positive projective tensor product.

IV-B6 Conditionals and $\mathtt{while}$ loops

We provide the semantics of conditionals, the semantics of $\mathtt{while}$ loops then follows exactly as in [5]: by first writing the fixpoint equation in terms of conditionals and then solving it using the side condition that $\llbracket\Gamma\rrbracket$ is order-complete.

Given a boolean test $\mathtt{\Gamma}\vdash e:\mathtt{bool}$ interpreted as an operator $\llbracket e\rrbracket:\llbracket\mathtt{\Gamma}\rrbracket\to\mathcal{M}2$, the order-completeness of $\llbracket\Gamma\rrbracket$ allows us to define the maps

[TABLE]

Proposition 19**.**

The maps $T_{e}$ and $F_{e}$ are additive and linear over $\mathbb{R}^{+}$.

Since regular ordered Banach spaces have a generating cone, we can uniquely extend $T_{e}$ and $F_{e}$ to the entire space $\llbracket\mathtt{\Gamma}\rrbracket$ and define the semantics of the conditional $\mathtt{\Gamma}\vdash\mathtt{if}\enskip e_{1}\enskip\mathtt{then}\enskip e_{2}\enskip\mathtt{else}\enskip e_{3}:\mathtt{T}$ as the operator

[TABLE]

To see why this definition makes sense we will briefly show that it recovers the semantics of [5]. In [5], $\llbracket\mathtt{\Gamma}\rrbracket$ is a measure space $\mathcal{M}X$ and $\llbracket e_{1}\rrbracket:\mathcal{M}X\to\mathcal{M}2$ is of the shape $\mathcal{M}b$ for a measurable map $b:X\to 2$ which specifies a measurable subset $B$ of $X$. We claim that $T_{e}:\mathcal{M}X\to\mathcal{M}X$ sends a probability measure $\mu$ to the measure $\mu_{B}$ defined by $\mu_{B}(A)=\mu(A\cap B)$, exactly as the operator $e_{B}$ of [5, 3.3.4]. By unravelling the definition we want to show that

[TABLE]

Note first that $\mu_{B}$ belongs to the set above, so it remains to show that it is its minimal element. Let $\nu$ also belong to this set, and let $A$ be a measurable set. We decompose $A$ as $A=(A\cap B)\uplus(A\cap B^{c})$. By definition

[TABLE]

Moreover we have

[TABLE]

For if we had $\nu(A\cap B)<\mu(A\cap B)$, then in order to keep $\mu(B)=\nu(B)$ we would need $\nu(A^{c}\cap B)>\mu(A^{c}\cap B)$, a contradiction with $\nu\leq\mu$. Thus $\mu_{B}\leq\nu$ as claimed and the semantics of $\mathtt{if}\enskip e_{1}\enskip\mathtt{then}\enskip e_{2}\enskip\mathtt{else}\enskip e_{3}:\mathtt{T}$ becomes the operator

[TABLE]

exactly as in [5].

IV-B7 $\mathtt{sampler}$ and $\mathtt{sample}$

are given a semantics which can be understood as generalisations of the unit and co-unit of the Giry monad [10] respectively. First we need the following easy result.

Theorem 20**.**

The semantics of every measure type is isometrically and monotonically embedded in a space of measures $\mathcal{M}X$.

We can now define the semantics of $\mathtt{sampler}$. Suppose we have $\llbracket e\rrbracket:\llbracket\mathtt{\Gamma}\rrbracket\to\llbracket\mathtt{T}\rrbracket$ and, by Th. 20, that $\llbracket\mathtt{T}\rrbracket$ is isometrically and monotonically embedded in the space $\mathcal{M}X$. Now consider the map $\eta:X\to\mathcal{M}X,x\mapsto\delta_{x}$ (which is not an operator) and define denotation of $\mathtt{sampler}(e)$ as the positive operator

[TABLE]

The semantics of sample works in the opposite direction. Suppose we have $\llbracket e\rrbracket:\llbracket\mathtt{\Gamma}\rrbracket\to\mathcal{M}\llbracket\mathtt{T}\rrbracket$ with $\llbracket\mathtt{T}\rrbracket$ isometrically and monotonically embedded in $\mathcal{M}X$, then each element of $\mathcal{M}\llbracket\mathtt{T}\rrbracket$ is also an element of $\mathcal{M}\mathcal{M}X$. We can define a map

[TABLE]

where we recall that $B^{+}(\mathcal{M}X)$ is the positive unit ball of the space $\mathcal{M}X$. This map clearly defines a positive operator, and in the case where $\rho$ is supported by the set of probability distributions, i.e. the shell $\{\mu\in(\mathcal{M}X)^{+}:\|\mu\|=1\}$ of the positive unit ball, $m_{X}$ coincides with the multiplication of the Giry monad. We are now ready to define

[TABLE]

We can now interpret the type of the small Gaussian inference program of § III-B3. In defining the semantics of built-in operations we saw that the semantics of $\mathtt{\mathtt{\emptyset}\vdash normal(0,1):\mathtt{M}\mathtt{real}}$ is the linear map $\mathbb{R}\to\mathcal{M}\mathcal{M}\mathbb{R}$ mapping $1$ to the Dirac delta over the normal distribution $\mathcal{N}(0,1)$. It follows that

[TABLE]

and by unravelling the definition we similarly find that

[TABLE]

which is the pushforward of $\mathcal{N}(0,1)$ by the kernel $\lambda x.~{}\mathcal{N}(x,1)$ described in (1). It follows that the output of the Gaussian inference program is interpreted as a linear operator

[TABLE]

We now turn to the semantics of the observe statement, which will show us what this operator actually is.

IV-B8 Semantics of observe

Assume that we have $\llbracket e\rrbracket:\llbracket(\mathtt{S},\mu)\rrbracket\to\llbracket\mathtt{T}\rrbracket$ with $\mathtt{S},\mathtt{T}$ of measure type, i.e. we can type $\llbracket e\rrbracket$ as a regular operator $(\mathcal{M}X)_{\mu}\to\mathcal{M}Y$.

We now make the assumption, which we justify in Th. 23 below, that $\llbracket e\rrbracket$ is positive. Under this assumption if $\nu\leq K\mu$, then $\llbracket e\rrbracket(\nu)\leq K\llbracket e\rrbracket(\mu)$, i.e. $\llbracket e\rrbracket$ restricts to an operator

[TABLE]

where $(\mathcal{M}X)_{\mu}^{UB}$ is the set of measures uniformly bounded by a multiple of $\mu$ (see [20]).

The semantics of $\mathtt{observe}(e)$ is then fundamentally contained in the Köthe dual operator

[TABLE]

It is not hard to check by using the Riesz Representation and Functional Representations natural transformations ($\mathrm{RR}$ and $\mathrm{FR}$ in Diagram (3)) that $\left((\mathcal{M}Y)_{\nu}^{UB}\right)^{\sigma}\simeq(\mathrm{L}_{p}(Y,\nu))^{\sigma}\simeq(\mathcal{M}Y)_{\nu}$, and thus $\llbracket e\rrbracket^{\sigma}$ can, modulo these isomorphisms, be typed as an operator

[TABLE]

which is what the typing rule for observe requires.

To illustrate how this semantics really implements the Bayesian inversion described in § II-A3, let us again consider our simple Gaussian inference program. The underlying Bayesian model is given by the probability kernel $\mathcal{N}(-,1):\mathbb{R}\to\mathscr{M}\mathbb{R}$ and the prior $\mathcal{N}(0,1)$ on $\mathbb{R}$. Together these define a $\mathbf{Krn}$-arrow $(\mathbb{R},\mathcal{N}(0,1))\to(\mathbb{R},\mathcal{N}(0,\sqrt{2}))$ which is implemented by the program

[TABLE]

whose denotation is the positive operator

[TABLE]

Using the same argument as above, we can restrict this operator as follows

[TABLE]

As stated above, all the information about the semantics of $\mathtt{observe(sample(normal(x,1)))}$ is contained in the Köthe dual of this operator, which, through the Riesz Representation and Functional Representations natural transformation, can be typed modulo isomorphism as

[TABLE]

Using the other half of diagram (3), that is to say the Radon-Nikodym and Measure Representation natural transformations ($\mathrm{RN}$ and $\mathrm{MR}$ in diagram 3), this operator is equal, modulo isomorphism, to the operator

[TABLE]

Here the Bayesian inverse of our original probability kernel appears explicitly, showing that our semantics indeed captures the notion of Bayesian inverse.

There is one final subtlety which we need to account for. Given $\llbracket e\rrbracket:\llbracket(\mathtt{S},\mu)\rrbracket\to\llbracket\mathtt{T}\rrbracket$, the typing rule for observe in fact makes the whole semantics described above parametric in a choice of measure absolutely continuous w.r.t. the prior $\mu$ (see Fig. 1). This is a simple technicality: morally and practically the parameter will always be set to the prior itself, in which case we get as output of the program the Köthe dual described by (12). Mathematically however, we can choose as input any $\nu\ll\mu$; the output operator is then defined by the following tortuous journey (similar to constructions in [20])

[TABLE]

where $i,j$ are the obvious inclusions.

Remark 21**.**

The semantics of observe via the Köthe dual is more general than a semantics in terms of Bayesian inversion/disintegration. Nothing prevents the introduction of ground types which stand for measurable spaces in which disintegrations do not exist. However, the Köthe dual will still exist. Thus our semantics is free of some of the ‘pointful’ technicalities surrounding the existence of disintegrations, and follow the ‘pointless’ perspective advocated in [13]. Similarly, we do not have to worry about the ambiguity cause by the fact that disintegrations are only defined up to a null set: the Köthe dual of an operator between regular ordered Banach spaces exists completely unambiguously.

IV-C Some properties of the semantics

The development of the semantics in the previous section has built-in soundness:

Theorem 22**.**

The semantics is sound w.r.t. to the typing rules of Fig. 1.

More importantly, we can extend [5, Th. 3.3.8] by a straightforward induction and show that:

Theorem 23**.**

The semantics of any program is a positive operator of norm $\leq 1$.

However another result of [5], namely that the denotation of a program is entirely determined by its action on point masses [5, Th. 6.1] does not hold any more. The reason is interesting and is worth a few words. It is immediate from the type system (Fig. 1) and the denotation of Bayesian types that the domain of the semantics of an observe statement may not contain any point masses at all. For example in the case of the Gaussian inference program of III-B3, this domain is $(\mathcal{M}\mathbb{R})_{\mathcal{N}(0,1)}$ which contains no point masses at all.

IV-D Comparison with semantics à la Scott.

There are interesting parallels to be drawn between our semantics and the Scott-Strachey semantics in terms of domains.

Looking at ground types first, it is worth noting that just like the flat domain functor turns a set (of integers for example) into a valid semantic object (a domain), so the functor $\mathcal{M}$ turns a measurable set into a valid semantic object (a regular ordered Banach space). Similarly, just like the flat domain functor can turn a partial map between sets into a total map between domains, so the functor $\mathcal{M}$ turns a partial measurable map into a linear operator. Partiality is encoded by the presence of the bottom element in the case of domain, and by the possibility to lose mass (i.e. get subdistributions) in the case of spaces of measures.

We do not know yet if the semantics of every program in our language is $\sigma$-order continuous, which would be the equivalent of Scott-continuous in our setting. The fundamental difference however is that our semantic category is not Cartesian closed, but monoidal closed.

Acknowledgment

The authors would like to thank Ilias Garnier for bringing [16] to their attention.

Appendix

IV-E Proofs

Proof of Proposition 1.

By the triangle inequality it is enough to reason about positive operators. The proof is by contradiction. Suppose that $f:U\to V$ is positive but not bounded, then we can find a sequence $\{x_{n}\}$ in $U$ with $\|x_{n}\|=1$ such that $\|f(x_{n})\|\geq n^{3}$. By the axioms R1 and R2 and the positivity of $f$ we can assume w.l.o.g. that $x_{n}\geq 0$. Since $\sum_{n=1}^{\infty}\frac{\|x_{n}\|}{n^{2}}$ converges, we have by completeness of Banach spaces that $\sum_{n=1}^{\infty}\frac{x_{n}}{n^{2}}$ converges to an element $x$ of $U$. Since every $x_{n}\geq 0$, it follows that $0\leq\frac{x_{n}}{n^{2}}\leq x$. Since positive operators are automatically monotone it follows from axiom R1 that:

[TABLE]

for every $n\in\mathbb{N}$, a contradiction. Thus $f$ is (norm-) bounded and therefore continuous. ∎

Sketch of the proof of Theorem 8.

The proof of [17, Th. 20.2, Cor. 20.3] also holds when the domain space is a regular ordered space because the only property of the domain being used is that the positive cone is generating. Thus the set $V^{\sim}$ of regular functionals on $V$ forms an order-complete Riesz space. Similarly, the proof of [18, Th. 4.74] holds when the domain space is a regular ordered space. Thus $V^{\sim}$ forms a Banach space as well. From [17, Th. 22.2] we know that the set $V^{\sigma}$ is a band in $V^{\sim}$. Now assume that $v_{n}$ is a Cauchy sequence in $V^{\sigma}$. We know that it converges to a $v\in V^{\sim}$. By [17, Th. 15.6] every norm convergent sequence in a Banach lattice has a subsequence converging in order. Let $u_{n}$ be this sequence, i.e. $u_{n}$ converges in order to $v$. Since $V^{\sim}$ is order complete we can further assume by the $\liminf$ construction that $u_{n}$ is increasing, i.e. $u_{n}\uparrow v$. But since $V^{\sigma}$ is band, it must follow that $v\in V^{\sigma}$ as desired. ∎

Proof of Theorem 10.

By Th. 25 $(\mathcal{M}X)_{\mu}\simeq\mathrm{L}_{1}(X,\mu)$ and $(\mathcal{M}Y)_{\nu}\simeq\mathrm{L}_{1}(Y,\nu)$. For each u\in\mathrm{L}_{1}(X,\mu)~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\mathrm{L}_{1}(Y,\nu), let $f_{u}\in\mathrm{L}_{1}(X,\mathrm{L}_{1}(Y,\nu),\mu)$ be the associated Bochner $\mu$-integrable function given by Th 27. This map in turns defines $\phi_{u}:X\times Y\to\mathbb{R}$ by $\phi_{u}(x,y)=f_{u}(x)(y)$. It is easy to see that $\phi_{u}\in\mathrm{L}_{1}(X\times Y,\mu\times\nu)$:

[TABLE]

By Bochner’s integrability theorem [7, 11.44] $f_{u}$ is Bochner $\mu$-integrable iff $\|f_{u}\|_{1}$ is Lebesgue integrable which shows that the integral above is finite. The fact that the operation $u\mapsto f_{u}\mapsto\phi_{u}$ is an isometry is obvious, and it has an inverse which associates to $\phi\in\mathrm{L}_{1}(X\times Y,\mu\times\nu)$ the Bochner $\mu$-integrable function $f_{\phi}:X\to\mathrm{L}_{1}(Y,\nu),x\mapsto\phi(x,\cdot)$. ∎

Proof of Theorem 11.

By Th. 28 \mathcal{M}{X}~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\mathcal{M}{Y} can be embedded in $\mathcal{M}{(X,\mathcal{M}{Y})}$. This allows us to define an embedding J:\mathcal{M}{X}~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\mathcal{M}{Y}\to\mathcal{M}{(X\times Y)} as follows: for each u\in\mathcal{M}{X}~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0pt\mathcal{M}{Y}, let $\mu_{u}$ denote the corresponding vector-valued measure in $\mathcal{M}{(X,\mathcal{M}{Y})}$, we can then define a measure on $X\times Y$ by putting for any rectangle $A\times B$ of $X\times Y$

[TABLE]

It is easy to check that $\|J\|=1$. ∎

Proof of Theorem 15.

By induction on the structure of the type. For ground type, it follows from the fact that their denotations are spaces of the shape $\mathcal{M}X$ which is always a Banach lattice (see § II-B3). The same argument holds in the inductive case if the outermost constructor is $\mathtt{M}$. For tensor product it follows from the fact that the positive projective tensor product of two Banach lattice is a Banach lattice [25]. Finally, any band in a Banach lattice is a Banach lattice (see § II-B4). ∎

Proof of Theorem 16.

By induction on the structure of the type. We start with the first layer of the grammar (9). Ground types are interpreted as spaces of the shape $\mathcal{M}X$ which are AL spaces, and thus order-complete by Th. 6. Bayesian types $(\mathtt{G},\mu)$ built from ground types are of the shape $\mathcal{M}X_{\mu}$, and thus isomorphic to an $\mathrm{L}_{1}$-space as was shown in Ex. 7. These are AL spaces, and are thus order-complete. The case of positive projective tensors of the shape $(\mathtt{G},\mu)\otimes(\mathtt{G},\mu)$ then follows from Th. 10. Finally, for the outer layer of the syntax, it is clear that whatever $\mathtt{S}$ is, since $\llbracket\mathtt{M}\mathtt{S}\rrbracket=\mathcal{M}\llbracket\mathtt{S}\rrbracket$, we get an order-complete space. The case of internal homs follows from the well-known result from the theory of Riesz space which states that if $V$ is an order-complete space, so is $[U,V]$ (for a net $\{T_{\alpha}\}$ in $[U,V]$, the join is defined at each $u\in U^{+}$ as $\bigvee_{\alpha}T_{\alpha}(u)$ by order-completeness of $V$, this is map is linear and extends to an operator $U\to V$ by the fact that regular ordered spaces have a generating cone, for more details see the proof of [18, Th. 1.18]). ∎

Proof of Theorem 17.

By induction on the structure of the types. For the base case note that all ground types are interpreted as spaces of the shape $\mathcal{M}G$, on which the evaluation function $ev_{G}:\mathcal{M}G\to\mathbb{R},\mu\mapsto\mu(X)$ is defined and is a strictly positive functional. The same clearly holds for types of the shape $\mathtt{M}\mathtt{T}$. Suppose now that $\llbracket\mathtt{T}\rrbracket$ has a strictly positive functional, then for any $\mu:\mathtt{T}$, since $\llbracket(\mathtt{T},\mu)\rrbracket$ is a subspace of $\llbracket\mathtt{T}\rrbracket$ is clearly inherits this strictly positive functional. Now assume both $\llbracket\mathtt{S}\rrbracket$ and $\llbracket\mathtt{T}\rrbracket$ have strictly positive functionals $\phi$ and $\psi$ respectively. Then the map $\llbracket\mathtt{S}\rrbracket\times\llbracket\mathtt{T}\rrbracket\to\mathbb{R}$ defined by $(t,s)\mapsto\phi(t)\phi(s)$ is bilinear and strictly positive. It follows that there exists a strictly positive functional \phi\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\psi on \llbracket\mathtt{S}\rrbracket\mathrel{\vbox{\offinterlineskip\halign{\hfil#\hfil\cr\scalebox{0.9}[0.3]{\triangle}\cr\scalebox{1.1}{\otimes}\cr}}}_{|\pi|}\llbracket\mathtt{T}\rrbracket=\llbracket\mathtt{S}\otimes\mathtt{T}\rrbracket. For internal homs we proceed as follows: since $[\llbracket\mathtt{S}\rrbracket,\llbracket\mathtt{T}\rrbracket]$ is generated by positive operators, let us first consider a strictly positive operator $T$ and define

[TABLE]

(note how this definition mimics the definition of the operator norm). Since $\phi$ is strictly positive, the set over which the supremum is taken is not empty, and since $T$ and $\psi$ are strictly positive, $0<\chi(T)<\infty$ [29, p. 545]. Moreover, since the supremum of a set of sums equals the sum of the suprema, $\chi(T+S)=\chi(T)+\chi(S)$, and similarly since scalar multiplication distributes over suprema $\chi(\lambda T)=\lambda\chi(T)$, it follows that $\chi$ is linear on strictly positive operators. We can then extend $\chi$ to all regular operators by putting $\chi(T):=\chi(T^{+})-\chi(T^{-})$. ∎

Proof of Proposition 19.

To see that it is additive, note first that since $\llbracket\mathtt{\Gamma}\rrbracket$ is order-complete it is in particular a Riesz space, and it therefore has the Riesz decomposition property [7, 8.9], which means that if $0\leq z\leq x+y$ there exist $0\leq z_{1},z_{2}$ such that $z_{1}+z_{2}=z$ and $z_{1}\leq x,z_{2}\leq y$. From this and the linearity of $\llbracket e\rrbracket$ it now follows easily that $T_{e}$ and $F_{e}$ are additive. The linearity poses no problem. ∎

Proof of Theorem 20.

By induction on the structure of the type, with the base case being tautological. Similarly, the case of Bayesian types and types of the shape $\mathtt{M}\mathtt{T}$ are also tautological. The case of tensor product types follows from Th. 11 and Th. 12 (the monotonicity of the embedding is obvious). ∎

Proof of Theorem 23.

The proof is by induction on the derivation tree of the program (see Fig. 1). The result holds trivially for constants, variables and subtyping. For built-in operations it follows from the fact that the pushforward operation has norm 1. For assignments, one can easily show that the norm of the strictly positive functional built in Th. 17 and used in defining the semantics has norm 1 and assignment therefore also has norm 1. The case of sequencing and sequential composition follows from the fact that the composition of operators of norm $\leq 1$ has norm $\leq 1$. For $\lambda$-abstraction, the result follows from the fact that the universal bilinear map has norm 1, and the result is trivial for function application. For conditionals it follows from the definition of $T_{e}$ and $F_{e}$ and the fact that the norm is monotone in a regular ordered Banach space. For while loops the result follows from [5]. Finally, for the observe statement the result follows from the fact that the dual of an operator of norm $\leq 1$ has norm $\leq 1$. ∎

IV-F Background material on measure theory and Lebesgue integration.

IV-F1 Measures

A measurable space is a set $X$ equipped with a collection $\mathcal{F}$ of subsets—called the measurable subsets—which can intuitively be understood as the observable parts of $X$, and are therefore also referred to as events in the probabilistic literature. The collection $\mathcal{F}$ of measurable sets must contain the empty set and be closed under complementation and countable union. It follows from the de Morgan laws that $\mathcal{F}$ is also closed under countable intersection. Any collection of subsets satisfying these closure properties is called a $\sigma$-algebra.777The “$\sigma$” in $\sigma$-algebra refers to “countable unions” in the same way as $F_{\sigma}$ sets are countable unions of closed sets in descriptive set theory, with $\sigma$ standing for the German Summe, union. A $\sigma$-algebra is thus an $\omega$-complete Boolean algebra of sets. An important example of a measurable space is the set $\mathbb{R}$ together with its Borel $\sigma$-algebra, the smallest $\sigma$-algebra containing the open subsets of $\mathbb{R}$ with the usual topology. In general, the Borel sets of a topological space are the smallest $\sigma$-algebra containing the open sets.

A signed measure on a $\sigma$-algebra $\mathcal{F}$ over $X$ is a map $\mu:\mathcal{F}\to[-\infty,+\infty]$ associating to every event $B$ a “weight” $\mu(B)$ satisfying (i) $\mu(\emptyset)=0$; (ii) $\mu$ can assume at most one of the values $-\infty$ and $+\infty$; and (iii) $\mu$ is $\sigma$-additive, that is, $\mu(\cup_{i=1}^{\infty}A_{i})=\sum_{i=1}^{\infty}\mu(A_{i})$ for any countable pairwise disjoint collection $(A_{i})_{i\in\mathbb{N}}$ of measurable sets. A signed measure is called a measure if it assumes values in $[0,\infty]$, a finite signed measure if it assumes values in $(-\infty,\infty)$ (equivalently, if $|\mu(X)|<\infty)$, and a probability measure if it is a measure and $\mu(X)=1$.

The study of signed measures can largely be reduced to the study of measures as the following result shows.

Theorem 24** (Hahn-Jordan decomposition [30, §5.6.1]).**

Every signed measure $\mu$ has a unique decomposition as a difference $\mu=\mu^{+}-\mu^{-}$ of two measures, at least one of which is finite.

The total variation measure of a signed measure is defined as

[TABLE]

and the total variation of $\mu$ is then defined as $|\mu|(X)$. If $\mu$ is a measure, then $\|\mu\|=\mu(X)$. From Th. 24 it we have

[TABLE]

The last measure-theoretical definition we need is the following: if $\mu,\nu$ are measures on a $\sigma$-algebra $\mathcal{F}$, $\mu$ is said to be absolutely continuous w.r.t. $\nu$, notation $\mu\ll\nu$, if $\mu(B)=0$ whenever $|\nu|(B)=0$.

A map $f:(X,\mathcal{F})\to(Y,\mathcal{G})$ between measurable spaces is called measurable if for every measurable subset $U\in\mathcal{G}$, $f^{-1}(U)\in\mathcal{F}$. Given a such a measurable map and a measure $\mu$ on $X$, one defines the pushforward measure $f_{*}(\mu)$ on $(Y,\mathcal{G})$ by $f_{*}(\mu)(B)=\mu(f^{-1}(B))$.

IV-F2 Lebesgue integration [30, Ch. 4]

Given a measurable space $(X,\mathcal{F})$, a simple function is any real-valued function $f:X\to\mathbb{R}$ expressible as a sum of the form

[TABLE]

where $\alpha_{i}\in\mathbb{R}$, $B_{i}\in\mathcal{F}$, and $\mathbbm{1}_{B_{i}}$ is the characteristic function of $B_{i}$. A simple function can be equivalently expressed in many ways, depending on the choice of $n$ and the $\alpha_{i}$ and $B_{i}$. Given a signed measure $\mu$ on $\mathcal{F}$, the integral of a simple function (13) w.r.t. to $\mu$ is given by

[TABLE]

It can be shown that (14) is independent of the specific representation (13). Note that (14) is linear as a function of $\mu$. It thus follows from Theorem 24 that integration w.r.t. to signed measures is uniquely determined by integration w.r.t. to measures. Similarly, (14) is linear and positive w.r.t. simple functions (positive simple functions have positive integrals). Since any simple function can be expressed as the difference of two positive simple functions, it follows that integrating simple maps w.r.t. to signed measures is completely determined by integrating positive simple maps w.r.t. to measures. In this spirit, given a positive measurable function $f:X\to\mathbb{R}^{+}$, we define

[TABLE]

For a general function $f:X\to\mathbb{R}$, we define $f^{+}(x)=\max(f(x),0)$ and $f^{-}(x)=-\min(f(x),0)$. Clearly $f^{+},f^{-}$ are positive and $f=f^{+}-f^{-}$. We now define the integral of $f$ w.r.t. to a signed measure $\mu$ as

[TABLE]

We shall return to the order-theoretic property of integration in § II-B. We conclude this summary of Lebesgue integration with one of the most important theorems in measure theory.

Theorem 25** (Radon-Nikodym [30, Th 5.5.4]).**

Let $(X,\mathcal{F})$ be a measurable space, $\mu$ a finite measure888The result holds more generally for any dominating $\sigma$-finite measure. We will only need finite signed measures in this paper. on $X$, and $\nu$ a finite signed measure on $\mathcal{F}$ such that $\nu\ll\mu$. Then there exists a measurable function $f:X\to\mathbb{R}$, unique up to a $\mu$-nullset, such that

[TABLE]

The function $f$ is called the Radon-Nikodym derivative of $\mu$ w.r.t. $\nu$ and is denoted $\frac{d\nu}{d\mu}$. It was shown in [14] that the Radon-Nikodym derivative defines a natural transformation.

IV-G Supplementary material on projective tensor products

IV-G1 Definition of a Banach space

We only consider real vector spaces in this paper. We therefore simply say ‘vector space’ with the understanding that the scalar field is $\mathbb{R}$. A Banach space is a vector space $V$ equipped with a norm $\|\cdot\|:V\to[0,\infty)$ such that $V$ is complete for the metric induced by the norm, i.e. such that every Cauchy sequence in $V$ has a limit in $V$. The vector spaces $\mathbb{R}^{n}$ equipped with the usual Euclidean norm are Banach space.

IV-G2 Bochner integration

One can generalise the ideas behind the Lebesgue integral to give a definition of the integral of a function taking its value in an arbitrary Banach space $V$. As in the case of the Lebesgue integral, we start with simple functions. Given a measurable space $(X,\mathcal{F})$, we generalise (13) and say that a function $f:X\to V$ is simple if is expressible a sum

[TABLE]

where $v_{i}\in V,B_{i}\in\mathcal{F}$. We define the Bochner integral of this simple function as

[TABLE]

that is to say the weighted average of the vectors $v_{i}$ with the weights $\mu(B_{i})$. We will always assume that a Banach space comes equipped with its Borel $\sigma$-algebra. To extend Bochner integration to arbitrary measurable functions $X\to V$ is impossible in general since $V$, unlike $\mathbb{R}$, may have a very high dimension. Intuitively, it is in general not possible to approximate a function $f:X\to V$ with simple ones if the dimensionality of $V$ is too high. We thus restrict the definition of the Bochner integral to a limited class of measurable functions taking values in a separable subspace. Formally, a function $f:X\to V$ is $\mu$-essentially separately valued if there exists $E\in\mathcal{F}$ and a separable subspace $Y\subseteq V$ such that $\mu(E^{c})=0$ and $f(E)\subseteq Y$. We can now state:

Theorem 26** (Pettis Measurability Theorem [23, Prop 2.15]).**

Let $(X,\mathcal{F})$ be a measurable space, let $\mu$ be a finite measure on $\mathcal{F}$, and let $f:X\to V$ be a function taking values in a Banach space. Then $f$ is Borel measurable and $\mu$-essentially separately valued iff there exists a sequence $(f_{n})$ of simple functions converging $\mu$-a.e. to $f$, in which case $f$ is said to be $\mu$-measurable.

A $\mu$-measurable function $f:X\to V$ will be called Bochner $\mu$-integrable if the sequence of simple functions $(f_{n})$ converging $\mu$-a.e. to $f$ also satisfies

[TABLE]

where the integral is the ordinary Lebesgue integral. The Bochner integral of $f$ is the defined as the vector

[TABLE]

Following the example of Lebesgue integration, we define Lebesgue-Bochner space $\mathrm{L}_{1}(X,V,\mu)$ as the Banach space of (equivalence classes) of Bochner $\mu$-integrable functions $f:X\to V$ equipped with the norm

[TABLE]

Theorem 27** ([23] Ex. 2.19).**

\mathrm{L}_{1}(X,\mu)~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptV\simeq\mathrm{L}_{1}(X,V,\mu)* the space of Bochner $\mu$-integrable maps $f:X\to V$.*

IV-G3 Vector-valued measures

One can also characterise projective tensor products of the shape \mathcal{M}(X)~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptV. For this we need a couple of definitions which generalise the notion of measure to Banach spaces. Given a measurable space $(X,\mathcal{F})$ and a Banach space $V$, a $V$-valued vector measure on $\mathcal{F}$ is a $\sigma$-additive map $\mu:\mathcal{F}\to V$, where $\sigma$-additivity means that if $A_{i}$ is a countable sequence of pairwise disjoint measurable sets, then the series $\sum_{i}\mu(A_{i})$ converges to $\mu(\bigcup_{i}A_{i})$. The variation norm of a $V$-valued measure is defined exactly like in the scalar-valued case:

[TABLE]

where the norm $\|\mu(A)\|$ is of course taken in $V$.

Vector-valued measures and Bochner integration are related by a Radon-Nikodym type construction which can briefly be described as follows. Given a scalar measure $\lambda\in\mathcal{M}(X,\mathcal{F})$ and a Bochner $\lambda$-integrable function $f:X\to V$, we can use $f$ as a ‘density’ and define a $V$-valued measure $\mu$ in the obvious way by

[TABLE]

The notion of absolute continuity for vector-valued measures is defined as follows: a $V$-valued measure $\mu$ on $(X,\mathcal{F})$ is absolutely continuous w.r.t. to a scalar measure $\lambda\in\mathcal{M}(X,\mathcal{F})$ is $\lambda(E)=0$ implies $\mu(E)=0$.999Note that $\mu(E)=0$ is the zero of $V$ whilst $\lambda(E)=0\in\mathbb{R}$. It is now natural to ask whether the Radon-Nikodym Theorem generalises to vector-valued measures and Bochner integrable densities. The answer is usually negative (see [23, Ex. 5.13] for an example), and we therefore start from (15) to justify the following definition: a $V$-valued measure $\mu$ on $(X,\mathcal{F})$ has the Radon-Nikodym property if it has bounded variation and if for every finite scalar measure $\lambda\in\mathcal{M}(X,\mathcal{F})$ there exists a Bochner $\lambda$-integrable function $f:X\to V$ such that (15) holds. We can now state a characterisation of tensor products of the shape \mathcal{M}X~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptV.

Theorem 28** ([23] Th. 5.22).**

\mathcal{M}{X}~{}\widehat{\scalebox{1.1}{\otimes}}_{\pi}\hskip 2.0ptY* is isometrically isomorphic to the Banach space of $Y$-valued measures with the Radon-Nikodym property together with the variation norm.*