On the geometry of Bayesian inference

TL;DR

This paper introduces a geometric framework for Bayesian inference that quantifies agreement between priors, likelihoods, and posteriors, providing new tools for assessing compatibility and sensitivity in Bayesian analysis.

Contribution

It presents a novel geometric interpretation of Bayesian inference, defining measures of compatibility and sensitivity based on inner products and correlation-like metrics.

Findings

Introduces a geometric measure of compatibility similar to Pearson correlation.

Provides estimators for geometric quantities from posterior simulations.

Illustrates methods with real-world examples on drug usage, insect morphology, and cancer data.

Abstract

We provide a geometric interpretation to Bayesian inference that allows us to introduce a natural measure of the level of agreement between priors, likelihoods, and posteriors. The starting point for the construction of our geometry is the simple observation that the marginal likelihood can be regarded as an inner product between the prior and the likelihood. A key concept in our geometry is that of compatibility, a measure which is based on the same construction principles as Pearson correlation, but which can be used to assess how much the prior agrees with the likelihood, to gauge the sensitivity of the posterior to the prior, and to quantify the coherency of the opinions of two experts. Estimators for all the quantities involved in our geometric setup are discussed, which can be directly computed from the posterior simulation output. Some examples are used to illustrate our methods, including data related to on-the-job drug usage, midge wing length, and prostate cancer.

Equations

$$⟨g,h⟩={\int }_{\Theta }g(\theta )h(\theta )\mathrm{d}\theta ,g,h\in L_2(\Theta ).$$

$$p(\theta \mid y)=\frac{\pi (\theta )f(y\mid \theta )}{{\int }_{\Theta }\pi (\theta )f(y\mid \theta )\mathrm{d}\theta }=\frac{\pi (\theta )\ell (\theta )}{⟨\pi ,\ell ⟩},$$

$$\pi \angle \ell =\arccos \frac{⟨\pi ,\ell ⟩}{\Vert \pi \Vert \Vert \ell \Vert }.$$

$${\kappa }_{\pi ,\ell }=\frac{⟨\pi ,\ell ⟩}{\Vert \pi \Vert \Vert \ell \Vert }.$$

$$\mathcal{L}=\{g+kh:g,h\in L_2(\Theta ),k\in \mathbb{R}\}.$$

$$\{\lambda g+(1-\lambda )h:\lambda \in [0,1]\}.$$

$${\kappa }_{g,h}=\frac{⟨g,h⟩}{\Vert g\Vert \Vert h\Vert },g,h\in L_2(\Theta ).$$

$$⟨X,Y⟩={\int }_{\Omega }XY\mathrm{d}P,X,Y\in L_2(\Omega ,{\mathbb{B}}_{\Omega },P),$$

$$\Vert \pi {\Vert }^2=\Vert \pi -{\pi }_0{\Vert }^2+\Vert {\pi }_0{\Vert }^2.$$

$$\Vert \pi {\Vert }^2=1-H_{\pi }+o\{\pi ({\theta }^{\ast })-1\},$$

$$\Vert \pi (a,b)\Vert =\frac{\{B(2a-1,2b-1){\}}^{1/2}}{B(a,b)},$$

$$\Vert \ell \Vert =\left(\genfrac{}{}{0ex}{}{n}{n_1}\right)\{B\left(2n_1+1,2\left(n-n_1\right)+1\right){\}}^{1/2},$$

$${\kappa }_{\pi ,\ell }(a,b)=\frac{B(a^{\star },b^{\star })}{\{B(2a-1,2b-1)B(2n_1+1,2(n-n_1)+1){\}}^{1/2}},$$

$${\kappa }_{\pi ,p}(a,b)=\frac{B(a+a^{\star }-1,b+b^{\star }-1)}{\{B(2a-1,2b-1)B(2a^{\star }-1,2b^{\star }-1){\}}^{1/2}},$$

$$\begin{array}{r}{\kappa }_{{\pi }_1,{\pi }_2}(a_1,b_1,a_2,b_2)=\frac{B(a_1+a_2-1,b_1+b_2-1)}{\{B(2a_1-1,2b_1-1)B(2a_2-1,2b_2-1){\}}^{1/2}}.\end{array}$$

$$\kappa_{\pi_{1},\pi_{2}}=\frac{(\pi_{A}\pi_{B})^{1/2}}{\pi_{C}}\Bigr{|}_{\mu=0,\sigma^{2}=1},$$

$$\begin{array}{rl}{\pi }_A& =\text{NIG}({\mu }_1,2{\eta }_1,2{\nu }_1+3,{\nu }_1{\sigma }_1^2/({\nu }_1+3/2)),{\pi }_B=\text{NIG}({\mu }_2,2{\eta }_2,2{\nu }_2+3,{\nu }_2{\sigma }_2^2/({\nu }_2+3/2)),\\ {\pi }_C& =\text{NIG}(({\eta }_1{\mu }_1+{\eta }_2{\mu }_2)/({\eta }_1+{\eta }_2),{\eta }_1+{\eta }_2,{\nu }_1+{\nu }_2+3,\\ & \{{\nu }_1{\sigma }_1^2+{\nu }_2{\sigma }_2^2+{\eta }_1{\eta }_2({\mu }_1-{\mu }_2{)}^2/({\eta }_1+{\eta }_2)\}/({\nu }_1+{\nu }_2+3)).\end{array}$$

$$\{\begin{array}{l}\begin{array}{rl}{\mu }^{\star }& =(n\bar{Y}+{\eta }_0{\mu }_0)/(n+{\eta }_0),{\eta }^{\star }={\eta }_0+n,{\nu }^{\star }={\nu }_0+n,\\ {\sigma }^{2\star }& =\left\{{\nu }_0{\sigma }_0^2+\sum _{i=1}^n(Y_i-\bar{Y}{)}^2+{\eta }_{0}n{({\eta }^{\star })}^{-1}({\mu }_0-\bar{Y}{)}^2\right\}/{\nu }^{\star }.\end{array}\end{array}$$

$${\pi }_1^g(\lambda )=\frac{{\pi }_1(g^{-1}(\lambda ))}{g^{\prime }(g^{-1}(\lambda ))},{\pi }_2^g(\lambda )=\frac{{\pi }_2(g^{-1}(\lambda ))}{g^{\prime }(g^{-1}(\lambda ))},$$

$$\begin{array}{r}\frac{{\int }_{\Lambda }{\pi }_1^g(\lambda ){\pi }_2^g(\lambda )\mathrm{d}\lambda }{[{\int }_{\Lambda }\{{\pi }_1^g(\lambda ){\}}^2\mathrm{d}\lambda {\int }_{\Lambda }\{{\pi }_2^g(\lambda ){\}}^2\mathrm{d}\lambda {]}^{1/2}}=\frac{{\int }_{\Theta }{\pi }_1(\theta ){\pi }_2(\theta )/g^{\prime }(\theta )\mathrm{d}\theta }{[{\int }_{\Theta }\{{\pi }_1(\theta ){\}}^2/g^{\prime }(\theta )\mathrm{d}\theta {\int }_{\Theta }\{{\pi }_2(\theta ){\}}^2/g^{\prime }(\theta )\mathrm{d}\theta {]}^{1/2}}.\end{array}$$

$$\begin{array}{r}\hat{\theta }=\arg \underset{\theta \in (0,1)}{\max }f(n_1\mid \theta )=\frac{n_1}{n}=\frac{a^{\ast }-1}{a^{\ast }+b^{\ast }-2}=:m(a^{\ast },b^{\ast }),\end{array}$$

$$\begin{array}{r}\hat{\theta }=\arg \underset{\theta \in \Theta }{\max }f(y\mid \theta )=m_{\pi }({\alpha }_y^{\ast }):=\arg \underset{\theta \in \Theta }{\max }\pi (\theta \mid {\alpha }_y^{\ast }).\end{array}$$

$$g(\hat{\theta })=\hat{\lambda }=\arg \underset{\lambda \in \Lambda }{\max }\underset{\theta \in {\Theta }_{\lambda }}{\sup }f(y\mid \theta )=g(m_{\pi }({\alpha }_y^{\ast }))=\arg \underset{\lambda \in \Lambda }{\max }{\pi }^g(\lambda \mid {\alpha }_y^{\ast }),$$

$$f_{\theta }(y)=h(y)\exp \{{\eta }_{\theta }^{\mathrm{T}}T(y)-A({\eta }_{\theta })\},$$

$$\ell(\theta)=\bigg{[}\prod_{i=1}^{n}h(Y_{i})\bigg{]}\exp\bigg{\{}\eta_{\theta}^{{\mathrm{\scriptscriptstyle T}}}\sum_{i=1}^{n}T(Y_{i})-nA(\eta_{\theta})\bigg{\}}.$$

$$\pi (\theta \mid \tau ,n_0)=K(\tau ,n_0)\exp \{{\tau }^{\mathrm{T}}{\eta }_{\theta }-n_{0}A({\eta }_{\theta })\},$$

$$K(\tau,n_{0})=\bigg{[}\int_{\Theta}\exp\{\tau^{{\mathrm{\scriptscriptstyle T}}}\eta_{\theta}-n_{0}A(\eta_{\theta})\}\,\mathrm{d}\theta\bigg{]}^{-1}.$$

$$\{\begin{array}{l}{\kappa }_{\pi ,\ell }(\tau ,n_0)=\frac{\{K(2\tau ,2n_0)K(2{\sum }_{i=1}^{n}T(Y_i),2n){\}}^{1/2}}{K(\tau +{\sum }_{i=1}^{n}T(Y_i),n_0+n)},\\ {\kappa }_{\pi ,p}(\tau ,n_0)=\frac{\{K(2\tau ,2n_0)K(2\{\tau +{\sum }_{i=1}^{n}T(Y_i)\},2\{n_0+n\}){\}}^{1/2}}{K(2\tau +{\sum }_{i=1}^{n}T(Y_i),2n_0+n)},\\ {\kappa }_{p,\ell }(\tau ,n_0)=\frac{\{K(2\{\tau +{\sum }_{i=1}^{n}T(Y_i)\},2\{n_0+n\})K(2{\sum }_{i=1}^{n}T(Y_i),2n){\}}^{1/2}}{K(\tau +2{\sum }_{i=1}^{n}T(Y_i),n_0+2n)},\end{array}$$

$$\pi\bigg{(}\theta\mid\sum_{i=1}^{n}T(Y_{i}),n\bigg{)},$$

$$\pi (\eta \mid a,b)=\frac{1}{B(a,b)}\exp \{a\eta -(a+b)\log (1+exp(\eta ))\}.$$

Full text

keywords: Bayesian inference; Geometry; Harmonic mean estimator; Hilbert spaces; Marginal likelihood.

††footnotetext: Miguel de Carvalho is Assistant Professor of Statistics, School of Mathematics, The University of Edinburgh, UK (e-mail: miguel.decarvalho@ed.ac.uk). Garritt L. Page is Assistant Professor of Statistics, Department of Statistics, Brigham Young University, Provo, Utah (e-mail: page@stat.byu.edu). Bradley J. Barney is Visiting Assistant Professor of Statistics, Department of Statistics, Brigham Young University, Provo, Utah (e-mail: barney@stat.byu.edu). We thank the Editor, the Associate Editor, and a Reviewer for insighful comments on a previous version of the paper. We extend our thanks to J. Quinlan for research assistantship and discussions, and to V. I. de Carvalho, A. C. Davison, D. Henao, W. O. Johnson, A. Turkman, and F. Turkman for constructive comments. The research was partially supported by Fondecyt 11121186 and 11121131 and by FCT (Fundação para a Ciência e a Tecnologia) through UID/MAT/00006/2013.

1 Introduction

Assessing the influence that prior distributions and/or likelihoods have on posterior inference has been a topic of research for some time. One commonly used ad-hoc method suggests fitting a Bayes model using a few competing priors, then visually (or numerically) assessing changes in the posterior as a whole or using some pre-specified posterior summary. More rigorous approaches have also been developed. Lavine (1991) developed a framework to assess sensitivity of posterior inference to sampling distribution (likelihood) and the priors. Berger (1991) introduced the concept of Bayesian robustness which includes perturbation models (see also Berger and Berliner 1986). More recently, Evans and Jang (2011) have compared information available in two competing priors. Related to this work, Gelman et al. (2008) advocates the use of so-called weakly informative priors that purposely incorporate less information than available as a means of regularizing. Work has also been dedicated to the so-called prior–data conflict (Evans and Moshonov, 2006; Walter and Augustin, 2009; Al Labadi and Evans, 2016). Such conflict can be of interest in a wealth of situations, such as for evaluating how much prior and likelihood information are at odds at the node level in a hierarchical model (see Scheel, Green and Rougier, 2011, and references therein). Regarding sensitivity of the posterior distribution to prior specifications, Lopes and Tobias (2011) provide a fairly accessible overview.

We argue that a geometric representation of the prior, likelihood, and posterior distribution encourages understanding of their interplay. Considering Bayes methodologies from a geometric perspective is not new, but none of the existing geometric perspectives has been designed with the goal of providing a summary on the agreement or impact that each component of Bayes theorem has on inference and predictions. Aitchison (1971) used a geometric perspective to build intuition behind each component of Bayes theorem, Shortle and Mendel (1996) used a geometric approach to draw conditional distributions in arbitrary coordinate systems, and Agarawal and Daumé (2010) argued that conjugate priors of posterior distributions belong to the same geometry giving an appealing interpretation of hyperparameters. Zhu, Ibrahim and Tang (2011) defined a manifold on which a Bayesian perturbation analysis can be carried out by perturbing data, prior and likelihood simultaneously, and Kurtek and Bharath (2015) provide an elegant geometric construction which allows for Bayesian sensitivity analysis based on the so-called $\epsilon$-compatibility class and on comparison of posterior inferences using the Fisher–Rao metric.

In this paper, we develop a geometric setup along with a set of metrics that can be used to provide an informative preliminary ‘snap-shot’ regarding comparisons between prior and likelihood (to assess the level of agreement between prior and data), prior and posterior (to determine the influence that prior has on inference), and prior versus prior (to compare ‘informativeness’—i.e., a density’s peakedness—and/or congruence of two competing priors). To this end, we treat each component of Bayes theorem as an element of a geometry formally constructed using concepts from Hilbert spaces and tools from abstract geometry. Because of this, it is possible to calculate norms, inner products, and angles between vectors. Not only do each of these numeric summaries have intuitively appealing individual interpretations, but they may also be combined to construct a unitless measure of compatibility, which can be used to assess how much the prior agrees with the likelihood, to gauge the sensitivity of the posterior to the prior, and to quantify the coherency of the opinions of two experts. Estimating our measures of level of agreement is straightforward and can actually be carried out within an MCMC algorithm. An important advantage of our setting is that it offers a direct link to Bayes theorem, and a unified treatment that can be used to assess the level of agreement between priors, likelihoods, and posteriors—or functionals of these. To streamline the illustration of ideas, concepts, and methods we reference the following example (Christensen et al., 2011, pp. 26–27) throughout the article.

On-the-job drug usage toy example

Suppose interest lies in estimating the proportion $\theta\in[0,1]$ of US transportation industry workers that use drugs on the job. Suppose $n=10$ workers were selected and tested with the 2nd and 7th testing positive. Let ${y}=(Y_{1},\dots,Y_{n})$ with $Y_{i}=1$ denoting that the $i$th worker tested positive and $Y_{i}=0$ otherwise. Let $Y_{i}\mid\theta\overset{\text{iid}}{\sim}\text{Bern}(\theta)$, for $i=1,\ldots,n$, and $\theta\sim\text{Beta}(a,b)$, for $a,b>0$. Then, $\theta\mid{{y}}\sim\text{Beta}(a^{\star},b^{\star})$ with $a^{\star}=n_{1}+a$ and $b^{\star}=n-n_{1}+b$, where $n_{1}=\sum_{i=1}^{n}Y_{i}$.

Some natural questions our treatment of Bayes theorem will answer are: How compatible is the likelihood with this prior choice? How similar are the posterior and prior distributions? How does the choice of $\text{Beta}(a,b)$ compare to other possible prior distributions? While the drug usage example provides a recurring backdrop that we consistently call upon, additional examples are used throughout the paper to illustrate our methods.

In Section 2 we introduce the geometric framework in which we work and provide definitions and interpretations along with examples. Section 3 considers extensions of the proposed setup, Section 4 contains computational details, and Section 5 provides a regression example illustrating utility of our metric. Section 6 conveys some concluding remarks. Proofs are given in the supplementary materials.

2 Bayes geometry

2.1 A geometric view of Bayes theorem

Suppose the inference of interest is over a parameter ${\theta}$ which takes values on $\Theta\subseteq\mathbb{R}^{p}$. We consider the space of square integrable functions $L_{2}(\Theta)$, and use the geometry of the Hilbert space $\mathcal{H}=(L_{2}(\Theta),\langle\cdot,\cdot\rangle)$, with inner-product

[TABLE]

The fact that $\mathcal{H}$ is a Hilbert space is often known in mathematical parlance as the Riesz–Fischer theorem; for a proof see Cheney (2001, p. 411). Borrowing geometric terminology from linear spaces, we refer to the elements of $L_{2}(\Theta)$ as vectors, and assess their ‘magnitudes’ through the use of the norm induced by the inner product in (1), i.e., $\|\cdot\|=(\langle\cdot,\cdot\rangle)^{1/2}$.

The starting point for constructing our geometry is the observation that Bayes theorem can be written using the inner-product in (1) as follows

[TABLE]

where $\ell({\theta})=f({y}\mid{\theta})$ denotes the likelihood, $\pi({\theta})$ is a prior density, $p({\theta}\mid{y})$ is the posterior density and $\langle\pi,\ell\rangle=\int_{\Theta}\ f({y}\mid{\theta})\pi({\theta})\,\mathrm{d}{\theta}$ is the marginal likelihood or integrated likelihood. The inner product in (1) naturally leads to considering $\pi$ and $\ell$ that are in $L_{2}(\Theta)$, which is compatible with a wealth of parametric models and proper priors. By considering $p$, $\pi$, and $\ell$ as vectors with different magnitudes and directions, Bayes theorem simply indicates how one might recast the prior vector so as to obtain the posterior vector. The likelihood vector is used to enlarge/reduce the magnitude and suitably tilt the direction of the prior vector in a sense that will be made precise below.

The marginal likelihood $\langle\pi,\ell\rangle$ is simply the inner product between the likelihood and the prior, and hence can be understood as a measure of agreement between the prior and the likelihood. To make this more concrete, define the angle measure between the prior and the likelihood as

[TABLE]

Since $\pi$ and $\ell$ are nonnegative, the angle between the prior and the likelihood can only be acute or right, i.e., $\pi\angle\hskip 1.42271pt\ell\in[0,90^{\circ}]$. The closer $\pi\angle\hskip 1.42271pt\ell$ is to $0^{\circ}$, the greater the agreement between the prior and the likelihood. Conversely, the closer $\pi\angle\hskip 1.42271pt\ell$ is to $90^{\circ}$, the greater the disagreement between prior and likelihood. In the pathological case where $\pi\angle\hskip 1.42271pt\ell=90^{\circ}$ (which requires the prior and the likelihood to have all of their mass on disjoint sets), we say that the prior is orthogonal to the likelihood. Bayes theorem is incompatible with a prior being orthogonal to the likelihood as $\pi\angle\hskip 1.42271pt\ell=90^{\circ}$ indicates that $\langle\pi,\ell\rangle=0$, thus leading to a division by zero in (2). Similar to the correlation coefficient for random variables in $L_{2}(\Omega,\mathbb{B}_{\Omega},P)$—with $\mathbb{B}_{\Omega}$ denoting the Borel sigma-algebra over the sample space $\Omega$—, our target object of interest is given by a standardized inner product

[TABLE]

The quantity $\kappa_{\pi,\ell}$ quantifies how much an expert’s opinion agrees with the data, thus providing a natural measure of the level of agreement between prior and data.

Before exploring (4) more fully by providing interpretations and properties we concretely define how the term ‘geometry’ will be used throughout the paper. The following definition of abstract geometry can be found in Millman and Parker (1991, p. 17).

Definition 1** (Abstract geometry).**

An abstract geometry $\mathcal{A}$ consists of a pair $\{\mathcal{P},\mathcal{L}\}$, where the elements of set $\mathcal{P}$ are designed as points, and the elements of the collection $\mathcal{L}$ are designed as lines, such that:

For every two points $A,B\in\mathcal{P}$, there is a line $l\in\mathcal{L}$. 2. 2.

Every line has at least two points.

Our abstract geometry of interest is $\mathcal{A}=\{\mathcal{P},\mathcal{L}\}$, where $\mathcal{P}=L_{2}(\Theta)$ and the set of all lines is

[TABLE]

Hence, in our setting points can be, for example, prior densities, posterior densities, or likelihoods, as long as they are in $L_{2}(\Theta)$. Lines are elements of $\mathcal{L}$, as defined in (5), so that for example if $g$ and $h$ are densities, line segments in our geometry consist of all possible mixture distributions which can be obtained from $g$ and $h$, i.e.,

[TABLE]

A related interpretation of two-component mixtures as straight lines can be found in Marriott (2002, p. 82).

Vectors in $\mathcal{A}=\{\mathcal{P},\mathcal{L}\}$ are defined through the difference of elements in $\mathcal{P}=L_{2}(\Theta)$. For example, let $g\in L_{2}(\Theta)$ and let $0\in L_{2}(\Theta)$. Then $g=g-0\in L_{2}(\Theta)$, and hence $g$ can be regarded both as a point and as a vector. If $g,h\in L_{2}(\Theta)$ are vectors then we say that $g$ and $h$ are collinear if there exists $k\in\mathbb{R}$, such that $g({\theta})=kh({\theta})$. Put differently, we say $g$ and $h$ are collinear if $g({\theta})\propto h({\theta})$, for all ${\theta}\in\Theta$.

For any two points in the geometry under consideration, we define their compatibility as a standardized inner product (with (4) being a particular case).

Definition 2** (Compatibility).**

The compatibility between points in the geometry under consideration is defined as

[TABLE]

The concept of compatibility in Definition 2 is based on the same construction principles as the Pearson correlation coefficient, which would be based however on the inner product

[TABLE]

instead of the inner product in (1). However, compatibility is defined for priors, posteriors, and likelihoods in $L_{2}(\Theta)$ equipped with the inner product (1), whereas Pearson correlation works with random variables in $L_{2}(\Omega,\mathbb{B}_{\Omega},P)$ equipped with the inner product (8). Our concept of compatibility can be used to evaluate how much the prior agrees with the likelihood, to measure the sensitivity of the posterior to the prior, and to quantify the level of agreement of elicited priors. As an illustration consider the following example.

Example 1**.**

Consider the following densities $\pi_{0}(\theta)=I_{(0,1)}(\theta)$, $\pi_{1}(\theta)=1/2I_{(0,2)}(\theta)$, $\pi_{2}(\theta)=I_{(1,2)}(\theta)$, and $\pi_{3}(\theta)=1/2I_{(1,3)}(\theta)$. Note that $\|\pi_{0}\|=\|\pi_{2}\|=1$, $\|\pi_{1}\|=\|\pi_{3}\|=\sqrt{2}/2$, and; further, $\kappa_{\pi_{0},\pi_{1}}=\kappa_{\pi_{2},\pi_{3}}=\sqrt{2}/2$, thus implying that $\pi_{0}\angle\pi_{1}=\pi_{2}\angle\pi_{3}=45^{\circ}$. Also, $\kappa_{\pi_{0},\pi_{2}}=0$ and hence $\pi_{0}\perp\pi_{2}$.

As can be observed in Example 1, $(\pi_{a}\angle\pi_{b})/90^{\circ}$ is a natural measure of distinctiveness of two densities. In addition, Example 1 shows us how different distributions can be associated to the same norm and angle. Hence, as expected, any Cartesian representation $(x,y)\mapsto(\|\cdot\|\cos(\cdot\angle\cdot),\|\cdot\|\sin(\cdot\angle\cdot))$, will only allow us to represent some features of the corresponding distributions, but will not allow us to identify the distributions themselves.

To build intuition regarding $\kappa_{\pi,\ell}$, we provide Figure 1, where $\ell$ is set to $\text{N}(0,1)$ while $\pi=\text{N}(m,{\sigma^{2}})$ varies according to $m$ and ${\sigma^{2}}$. Figure 1 (i) corresponds to fixing ${\sigma^{2}}=1$ and varying $m$ while in the right plot $m=0$ is fixed and $\sigma^{2}$ varies. Notice that in plot (i) $\kappa_{\pi,\ell}=0.1$ corresponds to distributions whose means are approximately 3 standard deviations apart while a $\kappa_{\pi,\ell}=0.9$ corresponds to distributions whose means are approximately 0.65 standard deviations apart. Connecting specific values of $\kappa$ to specific standard deviation distances between means seems like a natural way to quickly get a rough idea of relative differences between two distributions. In Figure 1 (ii) it appears that if both distributions are centered at the same value, then one distribution must be very disperse relative to the other to produce $\kappa$ values that are small (e.g., $\leq 0.1$). This makes sense as there always exists some mass intersection between the two distributions considered. Thus, $\kappa_{\pi,\ell}$—to which we refer as compatibility—can be regarded as a measure of the level of agreement between prior and data. Some further comments regarding our geometry are in order:

• •

Two different densities $\pi_{1}$ and $\pi_{2}$ cannot be collinear: If $\pi_{1}=k\pi_{2}$, then $k=1$, otherwise $\int\pi_{2}({\theta})\,\mathrm{d}{\theta}\neq 1$.

• •

A density can be collinear to a likelihood: If the prior is Uniform then $p({\theta}\mid{y})\propto\ell({\theta})$, and hence the posterior is collinear to the likelihood, i.e., in such a case the posterior simply consists of a renormalization of the likelihood.

• •

Two likelihoods can be collinear: Let $\ell$ and $\ell^{*}$ be the likelihoods based on observing ${y}$ and ${y}^{*}$, respectively. The strong likelihood principle states that if $\ell({\theta})=f({\theta}\mid{y})\propto f({\theta}\mid{y}^{*})=\ell^{*}({\theta})$, then the same inference should be drawn from both samples (Berger and Wolpert, 1988). According to our geometry, this would mean that likelihoods with the same direction yield the same inference.

As a final comment on reparametrizations of the model, interpretations of compatibility should keep a fixed parametrization in mind. That is, we do not recommend comparing prior–likelihood compatibility for models with different parametrizations. Further comments on reparametrizations will be given below in Sections 2.3, 2.4, and 3.2.

2.2 Norms and their interpretation

As $\kappa_{\pi,\ell}$ is comprised of function norms, we dedicate some exposition to how one might interpret these quantities. We start by noting that in some cases the norm of a density is linked to the variance, as can be seen in the following example.

Example 2**.**

Let $U\sim\text{Unif}(a,b)$ and let $\pi({u})=(b-a)^{-1}I_{(a,b)}({u})$ denote its corresponding density. Then, it holds that $\|\pi\|={1/(12\sigma_{U}^{2})^{1/4}}$, where the variance of $U$ is ${\sigma^{2}_{U}=1/12(b-a)^{2}}$. Next, consider a Normal model $X\sim\text{N}(\mu,{\sigma^{2}_{X}})$ with known variance $\sigma^{2}_{X}$ and let $\phi$ denote its corresponding density. It can be shown that $\|\phi\|=\{\int_{\mathbb{R}}\phi^{2}(x;\mu,{\sigma^{2}_{X}})\,\mathrm{d}\mu\}^{1/2}={1/(4\pi\sigma_{X}^{2})^{1/4}}$ which is a function of ${\sigma_{X}^{2}}$.

The following proposition explores how the norm of a general prior density, $\pi$, relates with that of a Uniform density, $\pi_{0}$.

Proposition 1**.**

Let $\Theta\subset\mathbb{R}^{p}$ with $\lambda(\Theta)<\infty$ where $\lambda$ denotes the Lebesgue measure. Consider $\pi$ : $\Theta\to[0,\infty)$ a probability density with $\pi\in L_{2}(\Theta)$ and let $\pi_{0}$ denote a Uniform density on $\Theta$, then

[TABLE]

Since $\|\pi_{0}\|^{2}$ is constant, $\|\pi\|^{2}$ increases as $\pi$’s mass becomes more concentrated (or less Uniform). Thus, as can be seen from (9), $\|\pi\|$ is a measure of how much $\pi$ differs from a Uniform distribution over $\Theta$. This interpretation cannot be applied to $\Theta$’s that do not have finite Lebesgue measure as there is no corresponding proper Uniform distribution. Nonetheless, the notion that the norm of a density is a measure of its peakedness may be applied whether or not $\Theta$ has finite Lebesgue measure. To see this, evaluate $\pi(\theta)$ on a grid $\theta_{1}<\cdots<\theta_{D}$ and consider the vector $p=(\pi_{1},\dots,\pi_{D})$, with $\pi_{d}=\pi(\theta_{d})$ for $d=1,\ldots,D$. The larger the norm of the vector $p$, the higher the indication that certain components would be far from the origin—that is, $\pi(\theta)$ would be peaking for certain $\theta$ in the grid. Now, think of a density as a vector with infinitely many components (its value at each point of the support) and replace summation by integration to get the $L_{2}$ norm. Therefore, $\|\cdot\|$ can be used to compare the ‘informativeness’ of two competing priors with $\|\pi_{1}\|<\|\pi_{2}\|$ indicating that $\pi_{1}$ is less informative.

Further reinforcing the idea that the norm is related to the peakedness of a distribution, there is an interesting connection between $\|\pi\|$ and the (differential) entropy (denoted by $H_{\pi}$) which is described in the following proposition.

Proposition 2**.**

Suppose $\pi\in L_{2}(\Theta)$ is a continuous density on a compact $\Theta\subset\mathbb{R}^{p}$, and that $\pi({\theta})$ is differentiable on $\emph{int}(\Theta)$. Let $H_{\pi}=-\int_{\Theta}\pi({\theta})\log\pi({\theta})\,\emph{d}{\theta}$. Then, it holds that

[TABLE]

for some ${\theta}^{*}\in\emph{int}(\Theta)$.

The expansion in (10) hints that the norm of a density and the entropy should be negatively related, and hence as the norm of a density increases, its mass becomes more concentrated. In terms of priors, this suggests that priors with a large norm should be more ‘peaked’ relative to priors with a smaller norm. Therefore, the magnitude of a prior appears to be linked to its peakedness (as is demonstrated in (9) and in Example 2). While this might also be viewed as ‘informativeness,’ the $\text{Beta}(a,b)$ density has a higher norm if $(a,b)\in(1/2,1)^{2}$ than if $a=b=1$, possibly placing this interpretation at odds with the notion that $a$ and $b$ represent ‘prior successes’ and ‘prior failures’ in the Beta–Binomial setting. As will be further discussed in Section 2.5, a reviewer recognized that this seeming paradox is a consequence of the parameterization employed and is avoided when using the log-odds as the parameter.

As can be seen from (10), the connection between entropy and $\|\pi\|$ is an approximation at best. Just as a first-order Taylor expansion provides a poor polynomial approximation for points that are far from the point under which the expansion is made, the expansion in (10) will provide a poor entropy approximation when $\pi$ is not similar to a standard Uniform-like distribution $\pi_{0}$. However, since $\|\pi_{0}\|^{2}=1-H_{\pi_{0}}$, the approximation is exact for a standard Uniform-like distribution. We end this discussion by noting that integrals related to $\|\pi\|^{2}$ also appear in physical models on $L_{2}$-spaces and they are usually interpreted as the total energy of a physical system (Hunter and Nachtergaele, 2005, p. 142), and there is considerable frequentist literature on the estimation of the integrated square of a density (see Giné and Nickl, 2008, and references therein). Now, to illustrate the information that $\|\cdot\|$ and $\kappa$ provide, we consider the example described in Section 1.

Example 3** (On-the-job drug usage toy example, cont. 1).**

From the example in the Introduction we have $\theta\mid{{y}}\sim\text{Beta}(a^{\star},b^{\star})$ with $a^{\star}=n_{1}+a=2+a$ and $b^{\star}=n-n_{1}+b=8+b$. The norm of the prior, posterior, and likelihood are respectively given by

[TABLE]

and $\|p(a,b)\|=\|\pi(a^{\star},b^{\star})\|$, with $a,b>1/2$, and

[TABLE]

where $B(a,b)=\int_{0}^{1}u^{a-1}(1-u)^{b-1}\,\mathrm{d}u$.

Figure 2 (i) plots $\|\pi(a,b)\|$ and Figure 2 (ii) plots $\|p(a,b)\|$ as functions of $a$ and $b$. We highlight the prior values $(a_{0},b_{0})=(3.44,22.99)$ which were employed by Christensen et al. (2011). Because prior densities with large norms will be more peaked relative to priors with small norms, $\|\pi(a_{0},b_{0})\|=2.17$ is more peaked than $\|\pi(1,1)\|=1$ (Uniform prior) indicating that $\|\pi(a_{0},b_{0})\|$ is more ‘informative’ than $\|\pi(1,1)\|$. The norm of the posterior for these same pairs is $\|p(a_{0},b_{0})\|=2.24$ and $\|p(1,1)\|=1.55$, meaning that the posteriors will have mass more concentrated than the corresponding priors. The lines found in Figure 2 (ii) represent boundary lines such that all $(a,b)$ pairs that fall outside of the boundary produce $\|\pi(a,b)\|>\|p(a,b)\|$ which indicates that the prior is more peaked than the posterior (typically an undesirable result). If we used an extremely peaked prior, say $(a_{1},b_{1})=(40,300)$, then we would get $\|\pi(a_{1},b_{1})\|=4.03$ and $\|p(40,300)\|=4.04$ indicating that the peakedness of the prior and posterior densities is essentially the same.

Considering $\kappa_{\pi,\ell}$, it follows that

[TABLE]

with $a^{\star}=n_{1}+a$ and $b^{\star}=n-n_{1}+b$. Figure 3 (i) plots values of $\kappa$ as a function of prior parameters $a$ and $b$ with $\kappa_{\pi,\ell}(a_{0},b_{0})\approx 0.69$ being highlighted indicating a great deal of agreement with the likelihood. In this example a lack of prior–data compatibility would occur (e.g., $\kappa_{\pi,\ell}\leq 0.1$) for priors that are very peaked at $\theta>0.95$ or for priors that place substantial mass at $\theta<0.05$.

The values of the hyperparameters $(a,b)$ which, according to $\kappa_{\pi,\ell}$, are more compatible with the data (i.e., those that maximise $\kappa$) are given by $(a^{*},b^{*})=(3,9)$ and are highlighted with a star (*****) in Figure 3 (i). In Section 2.4 we provide some connections between this prior and maximum likelihood estimators.

2.3 Angles between other vectors

As mentioned, we are not restricted to use $\kappa$ only to compare $\pi$ and $\ell$. Angles between densities, and between likelihoods and densities or even between two likelihoods are available. We explore these options further using the example provided in the Introduction.

Example 4** (On-the-job drug usage toy example, cont. 2).**

Extending Example 3 and (12) we calculate

[TABLE]

with $a^{\star}=n_{1}+a$ and $b^{\star}=n-n_{1}+b$; for $\pi_{1}\sim\text{Beta}(a_{1},b_{1})$ and $\pi_{2}\sim\text{Beta}(a_{2},b_{2})$,

[TABLE]

To visualize how the hyperparameters influence $\kappa_{\pi,p}$ and $\kappa_{\pi_{1},\pi_{2}}$ we provide Figures 3 (ii) and (iii). Figure 3 (ii) again highlights the prior used in Christensen et al. (2011) with $\kappa_{\pi,p}(a_{0},b_{0})\approx 0.95$; see solid dot ($\bullet$). This value of $\kappa_{\pi,p}$ implies that both prior and posterior are concentrated on essentially the same subset of $[0,1]$, indicating a large amount of agreement between them. Disagreement between prior and posterior takes place with priors concentrated on high probabilities of $\theta$ being greater than 0.8. In Figure 3 (iii), $\kappa_{\pi_{1},\pi_{2}}$ is largest when $\pi_{2}$ is close to $\text{Unif}(0,1)$ (the distribution of $\pi_{1}$) and gradually drops off as $\pi_{2}$ becomes more peaked and/or less symmetric.

In the next example, we use another data illustration to demonstrate the application of $\kappa$ to a two-parameter model.

Example 5** (Midge wing length data).**

Let $Y_{1},\dots,Y_{n}\mid\mu,\sigma^{2}\overset{\text{iid}}{\sim}\text{N}(\mu,\sigma^{2})$, and $\mu\mid\sigma^{2}\sim\text{N}(\mu_{0},\sigma^{2}/\eta_{0})$ and $\sigma^{2}\sim\text{IG}(\nu_{0}/2,\sigma_{0}^{2}\nu_{0}/2)$; we refer to this conjugate prior distribution as $\text{NIG}(\mu_{0},\eta_{0},\nu_{0},\sigma_{0}^{2})$. In comparing $\pi_{1}=\text{NIG}(\mu_{1},\eta_{1},\nu_{1},\sigma_{1}^{2})$ and $\pi_{2}=\text{NIG}(\mu_{2},\eta_{2},\nu_{2},\sigma_{2}^{2})$, $\kappa_{\pi_{1},\pi_{2}}$ may be expressed as,

[TABLE]

with

[TABLE]

Note that (13) (whose derivation can be found in Section 5.1 of the Supplementary Materials) may also be used to compute $\kappa_{\pi_{1},p}$, since $p=\text{NIG}(\mu^{\star},\eta^{\star},\nu^{\star},\sigma^{2\star})$, with

[TABLE]

Computation of $\kappa_{\pi_{1},\ell}$ also adheres to Equation (13) if $n>3$ and $\pi_{2}=\text{NIG}(\bar{Y},n,n-3,\sum_{i=1}^{n}{(Y_{i}-\bar{Y})^{2}}/(n-3))$ because then $\ell$ is collinear to $\pi_{2}$. Hoff (2009, pp. 72–76) applied this model to a dataset of nine midge wing lengths, where he set $\mu_{0}=1.9$, $\eta_{0}=1$, $\nu_{0}=1$, and $\sigma_{0}^{2}=0.01$, while $\bar{Y}=1.804$ and $\sum_{i=1}^{n}{(Y_{i}-\bar{Y})^{2}}\approx 0.135$. This yields $\kappa_{\pi,p}\approx 0.28$, and thus the agreement between the prior and posterior is not particularly strong. Figure 4 (i) displays $\kappa_{\pi,p}$, as a function of $\mu_{0}$ and $\sigma_{0}^{2}$ while fixing $\nu_{0}=1$ and $\eta_{0}=1$. To evaluate how $\kappa_{\pi,p}$ is affected by $\nu_{0}$ and $\eta_{0}$, the analogous plot is displayed as Figure 4 (ii) when these values are fixed at $\nu_{0}=6$ and $\eta_{0}=9$; these alternative values for $\nu_{0}$ and $\eta_{0}$ are those which allow the compatibility between the prior and likelihood to be maximised. It is apparent from Figure 4 that a larger $\sigma_{0}^{2}$ increases $\kappa_{\pi,p}$ substantially, and a simultaneous increase of $\nu_{0}$ and $\eta_{0}$ would further propel this increase.

Some comments on reparametrizations are in order. We focus on the case of compatibility between two priors with a single parameter, but the rationale below also applies to compatibility between a prior and posterior, and in multiparameter settings. Let $\theta_{1}\sim\pi_{1}$ and $\theta_{2}\sim\pi_{2}$; further, let $g(\theta)=\lambda$ be a monotone increasing function, with range $\Lambda$, and let

[TABLE]

be prior densities of the transformed parameters, $g(\theta_{1})$ and $g(\theta_{2})$. It thus follows that

[TABLE]

The version of compatibility discussed in this section is thus invariant to linear transformations of the parameter. A variant to be discussed in Section 3.2 is more generally invariant to monotone increasing transformations.

2.4 Max-compatible priors and maximum likelihood estimators

In Example 3, we briefly alluded to a connection between priors maximising prior–likelihood compatibility $\kappa_{\pi,\ell}$ (to be termed as max-compatible priors) and maximum likelihood (ML) estimators, on which we now elaborate. Below, we use the notation $\pi({\theta}\mid{\alpha})$ to denote a prior on ${\theta}\in\Theta$, with ${\alpha}\in{A}$ are hyperparameters, and where $\dim({A})=q$ and $\dim(\Theta)=p$. (Think of the Beta–Binomial model, where $\theta\in\Theta=(0,1)$, and ${\alpha}=(a,b)\in{A}=(0,\infty)^{2}$.)

Definition 3** (Max-compatible prior).**

Let ${y}\sim f(\,\cdot\mid{\theta})$, and let $\mathcal{P}=\{\pi({\theta}\mid{\alpha}):{\alpha}\in{A}\}$ be a family of priors for ${\theta}$. If there exists ${\alpha}^{*}_{{y}}\in{A}$, such that $\kappa_{\pi,\ell}({\alpha}^{*}_{{y}})=1$, the prior $\pi({\theta}\mid{\alpha}^{*}_{{y}})\in\mathcal{P}$ is said to be max-compatible, and ${\alpha}^{*}_{{y}}$ is said to be a max-compatible hyperparameter.

The max-compatible hyperparameter, ${\alpha}^{*}_{{y}}$, is by definition a random vector, and thus a max-compatible prior density is a random function. Geometrically, a prior is max-compatible if and only if it is collinear to the likelihood in the sense that $\kappa_{\pi,\ell}({\alpha}^{*}_{{y}})=1$ if and only if $\pi({\theta}\mid{\alpha}_{{y}}^{*})\propto f({y}\mid{\theta})$, for all ${\theta}\in\Theta$.

The following example suggests there could be a connection between the ML estimator of ${\theta}$ and the max-compatibility parameter ${\alpha}^{*}_{{y}}$.

Example 6** (Beta–Binomial).**

Let $n_{1}\mid\theta\sim\text{Bin}(n,\theta)$, and suppose $\theta\sim\text{Beta}(a,b)$. Here, $\mathcal{P}=\{\beta(\theta\mid a,b):(a,b)\in(1/2,\infty)^{2}\},$ with $\beta(\theta\mid a,b)=\theta^{a-1}(1-\theta)^{b-1}/B(a,b).$ It can be shown that the max-compatible prior is $\pi(\theta\mid a^{*},b^{*})=\beta(\theta\mid a^{*},b^{*})$, where $a^{*}=1+n_{1}$, and $b^{*}=1+n-n_{1}$, so that

[TABLE]

with $f(n_{1}\mid\theta)=\binom{n_{1}}{n}\theta^{n_{1}}(1-\theta)^{n-n_{1}}$.

A natural question is whether there always exists a function $m:{A}\to\Theta$, as in (14), linking the max-compatible parameter with the ML estimator? The following theorem addresses this.

Proposition 3**.**

Let ${y}\sim f(\,\cdot\mid{\theta})$, and let $\widehat{{\theta}}$ be the ML estimator of ${\theta}$. In addition, let $\mathcal{P}=\{\pi({\theta}\mid{\alpha}):{\alpha}\in{A}\}$ be a family of priors for ${\theta}$. If there exists a unimodal max-compatible prior, then

[TABLE]

Proposition 3 states that the mode of the max-compatible prior coincides with the ML estimator, and in Example 6, $m(a^{*},b^{*})=(a^{*}-1)/(a^{*}+b^{*}-2)$ is indeed the mode of a Beta prior. A comment on parametrizations is in order. A corollary to Proposition 3 is that, due to invariance of ML estimators, if $m_{\pi}({\alpha}^{*}_{{y}})$ is the mode of the max-compatible prior for $\theta$ and $g(\theta)=\lambda$ is a function, then $g(m_{\pi}({\alpha}^{*}_{{y}}))$ is the mode of the max-compatible prior of the transformed parameter $\pi^{g}(\lambda\mid\alpha^{*}_{y})$. Formally,

[TABLE]

with $\Theta_{\lambda}=\{\theta:g(\theta)=\lambda\}$ and where $\Lambda$ is the range of $g$.

The max-compatible prior is a ‘prior’ to the extent that it belongs to a family of priors, but it is basically a posterior distribution (it depends on the data). Also, there are some links between the max-compatible prior and Hartigan’s maximum likelihood prior (Hartigan, 1998), which will be clarified in Section 2.5.

2.5 Compatibility in the exponential family

We now consider compatibility in the exponential family with density

[TABLE]

for given functions $T$ and $h$, and with $A(\eta_{\theta})=\log[\int h(y)\exp\{\eta_{\theta}^{{\mathrm{\scriptscriptstyle T}}}T(y)\}\,\mathrm{d}y]<\infty$ denoting the so-called cumulant function. Given a random sample from an exponential family, $Y_{1},\ldots,Y_{n}\mid\theta\overset{\text{iid}}{\sim}f_{\theta}$, it follows that

[TABLE]

The conjugate prior is known to be

[TABLE]

where $\tau$ and $n_{0}$ are parameters, and

[TABLE]

The posterior density is $\pi(\theta\mid\tau+\sum_{i=1}^{n}T(Y_{i}),n_{0}+n)$, with $\pi(\theta\mid\tau,n_{0})$ defined as in (15); cf Diaconis and Ylvisaker (1979). In this context, compatibility can be expressed using normalizing constants from various members of the conjugate prior family as follows

[TABLE]

for $(\tau,n_{0})$ for which the normalizing constants in (17) are defined. The max-compatible prior in the exponential family is given by the following data-dependent prior

[TABLE]

with $\pi(\theta\mid\tau,n)$ as in (15). Special cases of the results in (17) and (18) were manifest for instance in (12), Example 4, and Example 6.

As pointed out by a reviewer, working with the canonical parametrization brings numerous advantages, especially when measuring compatibility. Since the parametrization of a model is arbitrary (and hence the interpretation of the parameter may be different for each model) it is desirable to work in terms of a parametrization that preserves the same meaning regardless of the model under consideration. For exponential families, a natural choice is the canonical parameter $\eta_{\theta}=\theta$. For one thing, the conjugate prior on the canonical parameter always exists under very general conditions (Diaconis and Ylvisaker, 1979). In contrast, the conjugate family for an alternative parametrization as defined in (15) can be empty; see Gutiérrez-Peña and Smith (1995, Example 1.2). In what follows, we revisit the Beta–Binomial setting and showcase yet another advantage of working with the canonical parametrization.

Example 7**.**

Let $\eta=\log\{\theta/(1-\theta)\}$ be the natural parameter of $\text{Bin}(n,\theta)$ and consider the prior for $\theta$ as $\text{Beta}(a,b)$. The conjugate prior for the natural parameter is

[TABLE]

It is readily apparent that

[TABLE]

More informative priors (i.e. larger values of $a$ and/or $b$) will always be more ‘peaked’ than less informative ones, and there is no need to constrain the range of values of the hyperparameters to the set $(1/2,\infty)$, as it was the case in (11). Finally, note that the max-compatible prior under the canonical parametrization is $\pi(\eta\mid n_{1},n-n_{1})$, whereas the max-compatible prior under the parametrization used earlier in Example 6 was $\beta(\theta\mid 1+n_{1},1+n-n_{1})$.

There are some links between the max-compatible prior introduced in Section 2.4 and Hartigan’s maximum likelihood prior (Hartigan, 1998). In the context of the exponential family, Hartigan’s maximum likelihood prior is a uniform distribution on the canonical parameter $\eta$. Equation (18) then implies that the max-compatible prior on the canonical parameter $\pi(\eta\mid\sum_{i=1}^{n}T(Y_{i}),n)$, can be regarded as a posterior derived from Hartigan’s maximum likelihood prior.

3 Extensions

3.1 Local prior–likelihood compatibility

In some cases, when assessing the level of agreement between prior and likelihood, integrating over $\Theta$ may not be feasible, but one can still assess the level of agreement over priors supported on a subset of the parameter space. Below $\Theta$ represents the parameter space and $\Pi$ denotes the support of the prior. More specifically, let $\pi$ be a prior supported on $\Pi=\{\theta:\pi(\theta)>0\}\subseteq\Theta$. We define local prior–likelihood compatibility as

[TABLE]

where $\langle\pi,\ell\rangle^{*}=\int_{\Pi}\pi(\theta)\ell(\theta)\,\mathrm{d}\theta$, $\|\ell\|^{*}=\{\int_{\Pi}\ell^{2}(\theta)\,\mathrm{d}\theta\}^{1/2}$, and $\|\pi\|^{*}=\{\int_{\Pi}\pi^{2}(\theta)\,\mathrm{d}\theta\}^{1/2}$. Note that

[TABLE]

and thus if $\Pi=\Theta$, then $\kappa^{*}_{\pi,\ell}=\kappa_{\pi,\ell}$. In practice, we recommend using standard likelihood–prior compatibility (4) instead of its local version (19), with the exception of situations for which the likelihood is square integrable over $\Pi$ but not over $\Theta$. To illustrate that (19) could be well defined even if (4) is not, suppose $Y\mid\mu,\sigma^{2}\sim\text{N}(\mu,\sigma^{2})$ with $\mu\sim\text{N}(m,s^{2})$ and $\sigma\sim\text{Unif}(a,b)$, for $0<a<b$. In this pathological single-observation case (4) would not be defined, while it follows that,

[TABLE]

Since (4) only assesses the level of agreement locally—that is, over $\Pi\subseteq\Theta$—the values of (4) and (19) are not directly comparable. A local $\kappa_{\ell,p}^{*}$ can be analogously defined to (19).

3.2 Affine-compatibility

We now comment on a version of our geometric setup where one no longer focuses directly on angles between priors, likelihoods, and posteriors, but on functions of these. Specifically, we consider the following measures of agreement,

[TABLE]

Some affine-compatibilities in (20) are Hellinger affinities (van der Vaart, 1998, p. 211), and thus have links with Kurtek and Bharath (2015) and Roos et al. (2015). Action does not always takes place at the Hilbert sphere, given the need of considering $\kappa_{\sqrt{\pi},\sqrt{\ell}}$. Local versions of prior–likelihood and likelihood–posterior affine-compatibility, $\kappa_{\sqrt{\pi},\sqrt{\ell}}$ and $\kappa_{\sqrt{\ell},\sqrt{p}}$, can be readily defined using the same principles as in Section 3.1.

It is a routine exercise to prove that max-compatible hyperparameters also maximise $\kappa_{\sqrt{\pi},\sqrt{\ell}}$, and thus all comments on Section 2.4 also apply to prior–likelihood affine-compatibility. In terms of affine-compatibility in the exponential family, following the same notation as in Section 2.5, it can be shown that

[TABLE]

with $K(\tau,n_{0})$ as defined in (16).

Affine-compatibility between priors and posteriors is invariant to monotone increasing parameter transformations, as a consequence of properties of the Hellinger distance (Roos and Held, 2011, p. 267). Affine-compatibility counterparts of all data examples are available from the supplementary materials; the conclusions are tantamount to the ones using compatibility.

4 Posterior and prior mean-based estimators of compatibility

In many situations closed form estimators of $\kappa$ and $\|\cdot\|$ are not available. This leads to considering algorithmic techniques to obtain estimates. As most Bayes methods resort to MCMC methods it would be appealing to express $\kappa_{\cdot,\cdot}$ and $\|\cdot\|$ as functions of posterior expectations and employ MCMC iterates to estimate them. For example, $\kappa_{\pi,p}$ can be expressed as

[TABLE]

where $E_{p}(\,\cdot\,)=\int_{\Pi}\cdot\,p({\theta}\mid{y})\,\mathrm{d}{\theta}$ is the expected value with respect to the posterior density. A natural Monte Carlo estimator would then be

[TABLE]

where ${\theta}^{b}$ denotes the $b$th MCMC iterate of $p({\theta}\mid{y})$. Consistency of such an estimator follows trivially by the ergodic theorem and the continuous mapping theorem, but there is an important issue regarding its stability. Unfortunately, (22) includes an expectation that contains $\ell({\theta})$ in the denominator and therefore (23) inherits the undesirable properties of the so-called harmonic mean estimator (Newton and Raftery, 1994). It has been shown that even for simple models this estimator may have infinite variance (Raftery et al. 2007), and has been harshly criticized for, among other things, converging extremely slowly. Indeed, as argued by Wolpert and Schmidler (2012, p. 655): “the reduction of Monte Carlo sampling error by a factor of two requires increasing the Monte Carlo sample size by a factor of $2^{1/\varepsilon}$, or in excess of $2.5\cdot 10^{30}$ when $\varepsilon=0.01$, rendering [the harmonic mean estimator] entirely untenable.”

An alternate strategy is to avoid writing $\kappa_{\pi,p}$ as a function of harmonic mean estimators and instead express it as a function of posterior and prior expectations. For example, consider

[TABLE]

where $E_{\pi}(\,\cdot\,)=\int_{\Pi}\cdot\,\pi({\theta})\,\mathrm{d}{\theta}$. Now the Monte Carlo estimator is

[TABLE]

where ${\theta}_{b}$ denotes the $b$th draw of ${\theta}$ from $\pi({\theta})$, which can also be sampled within the MCMC algorithm. Although representations (24) and (25) could in principle suffer from numerical instability for diffuse priors, they behave much better in practice than (22) and (23). To see this, Figure 5 contains running estimates of $\kappa_{\pi,p}$ using (23) and (25) for Example 3 with three prior parameter specifications, namely: $(a=1,b=1)$, $(a=2,b=1)$, and $(a=10,b=1)$; the true $\kappa_{\pi,p}$ for each prior specification is also provided. It is fairly clear that $\hat{\kappa}_{\pi,p}$ displays slow convergence and large variance, while $\tilde{\kappa}_{\pi,p}$ converges quickly.

The next proposition contains prior and posterior mean-based representations of geometric quantities that can be readily used for constructing Monte Carlo estimators.

Proposition 4**.**

Let $\pi$ be a prior supported on $\Pi=\{\theta:\pi(\theta)>0\}\subseteq\Theta$, with $\|\ell\|^{*}$ and $\kappa_{\pi,\ell}^{*}$ be defined as in (19), and let $E_{p}(\,\cdot\,)=\int_{\Pi}\cdot\,p({\theta}\mid{y})\,\mathrm{d}{\theta}$ and $E_{\pi}(\,\cdot\,)=\int_{\Pi}\cdot\ \pi({\theta})\,\mathrm{d}{\theta}$. Then,

[TABLE]

Similar derivations can be used to obtain posterior and prior mean-based estimators for affine-compatibility; see supplementary materials. In the next section we provide an example that requires the use of Proposition 4 to estimate $\kappa$ and $\|\cdot\|$.

5 Example: Regression shrinkage priors

5.1 Compatibility of Gaussian and Laplace priors

The linear regression model is ubiquitous in applied statistics. In vector form, the model is commonly written as

[TABLE]

where ${y}=(Y_{1},\dots,Y_{n})^{{\mathrm{\scriptscriptstyle T}}}$, ${X}$ is a $n\times p$ design matrix, ${\beta}$ is a $p$-vector of regression coefficients, and $\sigma^{2}$ is an unknown idiosyncratic variance parameter; the experiments below employ $\sigma\sim\text{Unif}(0,2)$. We consider Gaussian and Laplace prior distributions for ${\beta}$. As documented in Park and Casella (2008) and Kyung et al. (2010) ridge regression and $\beta_{j}\overset{\text{iid}}{\sim}\text{N}(0,\lambda^{2})$ produce the same regularization on ${\beta}$ while the lasso produces the same regularization on $\bm{\beta}$ as assuming $\beta_{j}\overset{\text{iid}}{\sim}\text{Laplace}(0,b)$ (where $\text{var}(\beta_{j})=2b^{2}$). Below, we use $\pi_{1}$ to denote a Gaussian prior and $\pi_{2}$ a Laplace. Further, we set $b=\sqrt{0.5\lambda^{2}}$ which ensures that $\text{var}_{\pi_{1}}(\beta_{j})=\text{var}_{\pi_{2}}(\beta_{j})=\lambda^{2}$ for all $j$.

5.2 Prostate cancer data example

We now consider the prostate cancer data example found in Hastie, Tibshirani and Friedman (2008, Section 3.4) to explore the ‘informativeness’ of and various compatibility measures for $\pi_{1}$ and $\pi_{2}$. In this example the response variable is the level of prostate-specific antigens measured on 97 males. Eight other clinical measurements (such as age and log prostate weight) were also measured and are used as covariates.

We first evaluate the ‘informativeness’ of the two priors by computing $\|\pi_{1}\|$ and $\|\pi_{2}\|$ and then their compatibility using $\kappa_{\pi_{1},\pi_{2}}$. All calculations employed Proposition 4 and results for a sequence of $\lambda^{2}$ values are provided in Figure 6. Focusing on the left plot of Figure 6 it appears that for small values of the $\lambda^{2}$, $\|\pi_{1}\|<\|\pi_{2}\|$, indicating that the Laplace prior is more peaked than the Gaussian. Thus, even though the Laplace has thicker tails, it is more ‘informative’ relative to the Gaussian. This corroborates the lasso penalization’s ability to shrink coefficients to zero (something ridge regulation lacks). As $\lambda^{2}$ increases the two norms converge as both spread their mass more uniformly. The right plot of Figure 6 depicts $\kappa_{\pi_{1},\pi_{2}}$ as a function of $\lambda^{2}$. When $\pi_{1}$ is centered at zero, then $\kappa_{\pi_{1},\pi_{2}}$ is constant over values of $\lambda^{2}$ which means that mass intersection when both priors are centered at zero is not influenced by tail thickness. Compare this to $\kappa$ values when $\pi_{1}$ is not centered at zero [i.e., $\pi_{1}\sim\text{MVN}(0.5{j},\lambda^{2}{I})$ or $\pi_{1}\sim\text{MVN}(2{j},\lambda^{2}{I})$]. For the former, $\kappa$ increases as intersection of prior and posterior mass increases. For the latter, $\lambda^{2}$ must be greater than two for there to be any substantial mass intersection as $\kappa_{\pi_{1},\pi_{2}}$ remains essentially at zero.

We now fit model (26) to the cancer data and use Proposition 4 to calculate various measures of compatibility. Without loss of generality we centered the ${y}$ so that ${\beta}$ does not include an intercept and standardized each of the eight covariates to have mean zero and standard deviation one. The results are available from Figure 7.

Focusing on the left plot of Figure 7 the small values of $\kappa_{\pi_{1},\ell}$ and $\kappa_{\pi_{2},\ell}$ indicate the existence of prior–data incompatibility. For small values of $\lambda^{2}$, $\kappa_{\pi_{1},\ell}>\kappa_{\pi_{2},\ell}$ indicating more compatibility between prior and data for the Gaussian prior. Prior–posterior compatibility ($\kappa_{\pi,p}$) is very similar for both priors with that for $\pi_{2}$ being slightly smaller when $\lambda^{2}$ is close to $10^{-4}$. The slightly higher $\kappa_{\pi,p}$ value for the Gaussian prior implies that it has slightly more influence on the posterior than the Laplace. Similarly, the Laplace prior seems to produce larger $\kappa_{\ell,p}$ values than that of the Gaussian prior and $\kappa_{\ell,p_{2}}$ approaches one quicker than $\kappa_{\ell,p_{1}}$ indicating a larger amount of posterior-data compatibility. Overall, it appears that the Gaussian prior has more influence on the resulting posterior distribution relative to the Laplace when updating knowledge via Bayes theorem. Similar conclusions as above would be reached by considering affine-compatibility; see supplementary materials.

6 Discussion

Bayesian inference is regarded from the viewpoint of the geometry of Hilbert spaces. The framework offers a direct connection to Bayes theorem, and a unified treatment that can be used to quantify the level of agreement between priors, likelihoods, and posteriors—or functions of these. The possibility of developing new probabilistic models, obeying the geometrical principles discussed here, offering alternative ways to recast the prior vector using the likelihood vector remains to be explored. In terms of high-dimensional extensions, one could anticipate that as the dimensionality increases, there is increased potential for disagreement between two distributions. Consequently, $\kappa$ would generally diminish as additional parameters are added, ceteris paribus, but a suitable offsetting transformation of $\kappa$ could result in a measure of ‘per parameter’ agreement.

Some final comments on related constructions are in order. Compatibility as set in Definition 2 includes as a particular case the measures of niche overlap in Slobodchikoff and Schulz (1980). Peakedness as discussed in here should not be confused with the concept of Birnbaum (1948). The geometry in Definition 1 has links with the so-called affine space and thus the geometrical framework discussed above is different but has many similarities with that of Marriott (2002) and also with the mixture geometry of Amari (2016). A key difference is that the latter approaches define an inner product with respect to a density which is the basis of the construction of the Fisher information while here we define it simply as the product of two functions in $L_{2}(\Theta)$, and connect the construction with Bayes theorem and with Pearson’s correlation coefficient. While here we deliberately focus on positive $g,h\in L_{2}(\Theta)$, the case of a positive $m\equiv g(\theta)+kh(\theta)\in L_{2}(\Theta)$—but with $g$ always positive and with $h$ negative on a part of $\Theta$—is of interest in itself, as well as the set values of $k$ ensuring positivity of $m$ for all $\theta$. Some further interesting setups would be naturally allowed by slightly extending our geometry, say to include ‘mixtures’ with negative weights. Indeed, the parameter $\lambda$ in (6) might in some cases be allowed to take some negative values while the resultant function is still positive; see Anaya-Izquierdo and Marriott (2007).

While not explored here, the use of compatibility as a means of assessing the suitability of a given sampling model, is a natural inquiry for future research.

Supplementary material: The online supplementary materials include the counterparts of the data examples in the paper for the case of affine-compatibility as introduced in Section 3.2, technical derivations, and proofs of propositions.