On the method of likelihood-induced priors

TL;DR

This paper introduces a method to construct priors directly from the likelihood function using information entropy, especially for exponential family models, linking likelihood properties to conjugate priors.

Contribution

It presents a four-step algorithm for deriving likelihood-induced priors based on information entropy and likelihood resolution, connecting likelihood form to prior construction.

Findings

Likelihood contains sufficient information for prior construction.

Likelihood-induced prior coincides with conjugate prior for exponential family models.

The method leverages coarse-graining and resolving power of likelihood.

Abstract

We demonstrate that the functional form of the likelihood contains a sufficient amount of information for constructing a prior for the unknown parameters. We develop a four-step algorithm by invoking the information entropy as the measure of uncertainty and show how the information gained from coarse-graining and resolving power of the likelihood can be used to construct the likelihood-induced priors. As a consequence, we show that if the data model density belongs to the exponential family, the likelihood-induced prior is the conjugate prior to the corresponding likelihood.

Equations

$$p\left(\mathbf{\theta }|\mathbf{d}\right)=\frac{p\left(\mathbf{d}|\mathbf{\theta }\right)p(\mathbf{\theta })}{p\left(\mathbf{d}\right)}$$

$$H_p=-\sum _{i=1}^{n}P(x_i)\ln \frac{P(x_i)}{M(x_i)}.$$

$$H_p=-{\int }_{\Omega }p(x)\ln \frac{p(x)}{m(x)}dx$$

$$L(\theta ;x_{1:n})=\prod _{i=1}^{n}p\left(x_i|\theta \right)$$

$$l(\theta ;x_{1:n})=\frac{1}{n}\ln \left\{L(\theta ;x_{1:n})\right\}=\frac{1}{n}\sum _{i=1}^n\ln \left\{p\left(x_i|\theta \right)\right\},$$

$$\hat{\theta }=\arg \underset{\theta }{\max }L(\theta ;x_{1:n})=\arg \underset{\theta }{\max }l(\theta ;x_{1:n}).$$

$$L(\theta ;x_{1:n})=\prod _{i=1}^{n}p\left(x_i|\theta \right)=\frac{1}{{\theta }^n}\exp \left(-\frac{1}{\theta }\sum _{i=1}^n{x}_i\right)$$

$$l(\theta ;x_{1:n})=-\ln \theta -{\theta }^{-1}\bar{x},$$

$$E_{\theta |\mathbf{x}}\left[l(\theta ;x_{1:n})\right]=-⟨\ln \theta ⟩-⟨{\theta }^{-1}⟩\bar{x}$$

$$S\left(\theta ,X\right)=\frac{1}{p\left(x|\theta \right)}\frac{\partial p\left(x|\theta \right)}{\partial \theta }=\frac{\partial \ln p\left(x|\theta \right)}{\partial \theta }.$$

$$E\left(S|\theta \right)={\int }_{\Omega }\frac{\partial \ln p\left(x|\theta \right)}{\partial \theta }p\left(x|\theta \right)dx=0$$

$$\mathcal{I}\left(\theta \right)={\int }_{\Omega }{\left(\frac{\partial \ln p\left(x|\theta \right)}{\partial \theta }\right)}^{2}p\left(x|\theta \right)dx\ge 0$$

$$m(\theta )d\theta \propto \sqrt{\det \mathcal{I}\left(\theta \right)}d\theta .$$

$$-{\int }_{\Omega }p(\theta |x)\ln \left(\frac{p(\theta |x)}{q(\theta |x)}\right)d\theta \le 0$$

$$-{\int }_{\Omega }p(\theta |x)\ln \left(\frac{q(\theta |x)}{m(\theta )}\right)d\theta \ge -{\int }_{\Omega }p(\theta |x)\ln \left(\frac{p(\theta |x)}{m(\theta )}\right)d\theta =H_p(x;m)$$

$${\int }_{\Omega }{f}_k(\theta )p(\theta |x)d\theta ={\int }_{\Omega }{f}_k(\theta )q(\theta |x)d\theta =F_k,\text{ for all }k=1,\dots ,r$$

$$q(\theta |x)=\frac{m(\theta )}{Z\left({\lambda }_1,\dots ,{\lambda }_r\right)}\exp \left(-\sum _{k=1}^r{\lambda }_k{f}_k\left(\theta \right)\right)$$

$$H_p(x;m)=-{\int }_{\Omega }p(\theta |x)\ln \left(\frac{p(\theta |x)}{m(\theta )}\right)d\theta \le \ln Z+\sum _{k=1}^r{\lambda }_k{F}_k$$

$$q\left(\theta \right)=\frac{1}{{\lambda }_2\Gamma \left({\lambda }_1\right)}{\left(\frac{\theta }{{\lambda }_2}\right)}^{-{\lambda }_1-1}\exp \left(-\frac{{\lambda }_2}{\theta }\right),\text{ where }{\lambda }_1,{\lambda }_2>0$$

$$p\left(\mathbf{x}|\mathbf{\eta }\right)=h\left(\mathbf{x}\right)\exp \left\{{\mathbf{\eta }}^T\cdot T\left(\mathbf{x}\right)\right\}g\left(\mathbf{\eta }\right)$$

$$L\left(\mathbf{\theta };{\mathbf{x}}_{1:n}\right)=\prod _{k=1}^{n}p\left(x_k|\mathbf{\eta }\right)=\left(\prod _{k=1}^{n}h\left(x_k\right)\right)g{\left(\mathbf{\eta }\right)}^n\exp \left\{{\mathbf{\eta }}^T\cdot T\left(\mathbf{x}\right)\right\}.$$

$$q\left(\mathbf{\theta }\right)\propto \sqrt{\det \mathcal{I}\left(\mathbf{\theta }\right)}g{\left(\mathbf{\eta }\right)}^{\gamma }\exp \left({\mathbf{\eta }}^T\cdot \mathbf{\lambda }\right).$$

Full text

1 Introduction

We argue that the functional form of the likelihood function is informative and in the absence of other types of information, it induces a prior on the unknown parameters of interest. Regardless of the data, the functional form of the likelihood depends on the design of experiment, the measurement methods and the model which is evaluated. In the following we demonstrate how this type of information can be used to construct priors for the parameters of interest.

2 Statement of the problem

In the Bayesian approach, the possible values of the unknown, say $\mathbf{\theta}$ are described by the posterior distribution $p\left(\boldsymbol{\theta}\left|\boldsymbol{d}\right.\right)$ through the Bayes’ theorem

[TABLE]

where $p\left(\boldsymbol{d}\left|\boldsymbol{\theta}\right.\right)$ is the likelihood, $p(\boldsymbol{\theta})$ is the prior and $p\left(\boldsymbol{d}\right)$ is the evidence, also known as the marginal likelihood. The likelihood is often well-determined by the process model and the data-generating process [Gre05]. The prior encodes ones belief about $\boldsymbol{\theta}$ before seeing the data. One of the challenges is to construct a prior that reflects ones state of knowledge before seeing the data. In many applications there are often little or no prior information available about $\boldsymbol{\theta.}$ Sometimes it is even difficult to interpret $\boldsymbol{\theta,}$ let alone describing the degree of belief. In such cases, there are several existing methods for constructing noninformative priors.

Uniform prior

Lack of information is modelled by bounded uniform distribution [Lap14].

Jeffreys prior

The prior is proportional to square root of the fisher information derived from the likelihood function [Jef98].

Reference prior

The idea is to obtain a prior that maximizes the expected gain of information provided by the data [BBS09].

Maximal data information prior

The approach is based on the information conservation principle [Zel71].

Although each one of these methods have their domain of usefulness, they implicitly ignore certain aspects of the information in a crucial way. The functional form of the likelihood contains some important information. In this sense, we are not ignorant about $\boldsymbol{\theta.}$ In the following, we demonstrate how the knowledge about the functional form of the likelihood imposes some sort of constraint on our belief, which in turn induces a prior on $\boldsymbol{\theta}$.

3 Measure of uncertainty

Uncertainty is a direct result of having multitude of choices. The assignment of probabilities to each choice reflects our belief. The certainty is only achieved in the limit when the number of choices is reduced to one. Therefore, in statistical inference, one aspires to achieve greater certainty by aggregation of choices. The purpose of the aggregation of any quantity is to create a less uncertain description without loss of essential information. In general, if $\boldsymbol{\theta}$ is the unknown, the information preserving aggregation is applied to possible values /functions of $\boldsymbol{\theta}$. Following this path of thought, two related questions present themselves,

What kind of information is preserved? 2. 2.

What does such information tell us about the probabilities of the events of interest?

In the following, we shall call the application of any information preserving aggregation as coarse-graining. In the probabilistic inference, the coarse-graining is conducted by applying the expectation operator with respect to a probability distribution on the quantity of interest. Moreover, since we are only interested on the coarse grainable quantities, such quantities will be referred to as conservation laws. In general, aggregation is destructive. If the aggregation is to be conservative then the distribution of the quantity of interest has to contain a sufficient amount of redundancy. More generally, regular structures contain a lot of redundancies whilst the random structures contain none. In the latter case, the aggregation would result in loss of information.

Since coarse-graining reduces the uncertainty, any study of such processes would depend on our ability to quantify uncertainty. That is, since uncertainty about a quantity is modelled by a probability distribution then we need a way to measure the amount of uncertainty described by a given probability distribution. This measure has already been defined in the closely related field of the information theory. Indeed, the fact that in statistical inference it is often possible to make inferences on the coarse-grained information bears the resemblance to the goal of information theory, which is to compress information such that it can be recovered exactly or approximately. In the information theory, Claude Shannon described three requirements which any measure of uncertainty should fulfill. However, in the present context, these requirements need a slight modification in order to meet our needs. Accordingly, these requirements can be stated as follows

Continuity:

The measure of uncertainty is a continuous function of probabilities. This means that small changes in the value of probabilities should only change the measure by a small amount.

Maximum

Maximum uncertainty is achieved when for all $i$ we have $P_{i}=M_{i},$ where $M_{i}$’s are determined by the resolving power of the available information.

Conservation of uncertainty:

Uncertainty in the fine grained-description is equal to the sum of uncertainties in the coarse-grained description and the amount concealed in each grain.

The maximum uncertainty is equivalent to lack of redundancy or compressible structure. That is, at the state of maximum uncertainty, aggregation of events would result in loss of information. At this state, the only thing one can do is to assign weights to events based on the resolving power of the measurements. This is a reasonable requirement in applications where measurements are part of inference. In the context of the transmission of messages, Shannon assumed that $M$ is the uniform distribution (see [Les14] for further insight on Shannon’s entropy and related subjects relevant to the present topic). The third requirement, the conservation of uncertainty, follows from the product and sum rules which impose constraints on how the probabilities of the events at the fine-grained level scale under coarse-graining. The functional that fulfils these three conditions is known as entropy and in the discrete case is given as

[TABLE]

This can be extended in the limit to the continuous case

[TABLE]

where $\Omega$ denotes the support of $p$ (see also [SJ80] for other axiomatic derivation). In order for this expression to be well-defined, $m$ has to dominate $p,$ that is, for almost all $x$ if $m(x)=0$ then $p(x)=0.$ In the discrete case, the positive function $M$ is the counting measure and in the continuous case $m$ is the Lebesgue measure. These measures can also be thought as mechanisms for assigning equal mass to regions of equal volume, which in the continuous case, guaranties that $H$ is invariant with respect to change of variables.

If $m$ is normalized then it follows from Jensen’s inequality that $H\leq 0,$ where the equality is achieved if and only if $p=m.$ Thus the uncertainty described by $p(x)$ is ranked relative to $m(x).$ In the following, we will only consider the continuous case ( 3) and it is assumed that $m$ is a probability density and all the integrals are proper and finite. The improper integrals are considered as the limit of proper integrals and are intended as useful approximations. In such cases, there is no guaranty that $m$ is normalizable and hence the upper limit of $H$ might be larger than zero or even without limit.

4 Conservation laws

The maximum likelihood method plays an important role in frequentist inference. The uncertainty reduction is conducted by choosing $\theta$ to be the global maximum of the likelihood. Since the logarithm is a monotonically increasing function, the maximum of the likelihood function is the same as the maximum of the corresponding average log-likelihood. For independent and identically distributed observations drawn from the data model density $p\left(x\left|\theta\right.\right),$ the likelihood and the average log-likelihood are defined as

[TABLE]

and

[TABLE]

respectively. Then the maximum likelihood estimate (MLE) is

[TABLE]

The use of MLE is justified in the limit when the number of observations grows to infinity. However, in reality we have only a finite number of observations. Therefore, the true maximum of the log-likelihood is not known and hence, in general, we can only talk about plausible candidates for $\theta.$ In the Bayesian context, the uncertainty about the average log-likelihood, $l(\theta;x_{1:n}),$ can be modelled by a probability distribution. For a given finite set of observations, $l(\theta;x_{1:n})$ depends only on $\theta$. Since we do not know the true value of $\theta,$ by appropriate coarse-graining with respect to $\theta,$ we can say something about the centre of mass of the distribution for $l(\theta;x_{1:n}).$ Often this coarse-graining reveals conservation laws, which as we shall see shortly, can be used to identify the family of distributions describing the uncertainty in $\theta.$

Let’s demonstrate this by an example in which the observations are generated from an exponential distribution with the parameter $\theta.$ Assume that the observations consists of $n$ independent data points, $x_{1:n}.$ Then the likelihood is given by

[TABLE]

and the average log-likelihood is

[TABLE]

where $\bar{x}$ is the arithmetic average of the observations. Clearly, uncertainty in $\theta$ results in uncertainty in $l(\theta;x_{1:n})$. Coarse-graining of $l(\theta;x_{1:n})$ with respect to $\theta$ results in a single number, which is the centre of mass for the distribution assigned to $l(\theta;x_{1:n}).$ The coarse-graining with respect to $\theta$ is conducted by taking the expectation with respect to $p\left(\theta|x_{1:n}\right),$ i.e.

[TABLE]

where the operator $\left\langle\cdot\right\rangle$ denotes the expectation taken with respect to $p\left(\theta|x_{1:n}\right).$ This closely follows the Bayesian philosophy that if one does not know the true value of $\theta$ then one should average over all its possible values. In the present example, it is evident that all the information from the distribution of $\theta$ relevant for determining $\left\langle l(\theta;x_{1:n})\right\rangle$ are summarized by the numbers $\left\langle\ln\theta\right\rangle$ and $\left\langle\theta^{-1}\right\rangle.$ This implies that the information about $E\left(l\right)$ induces a class of distributions for $\theta$ that conserve the expected values of $f_{1}(\theta)=\ln\theta$ and $f_{2}(\theta)=\theta^{-1}$ consistent with $E\left(l\right).$ In general, this class is very large. However, as we shall see, in the absence of any other information, these likelihood induced conservation laws can be used to identify the parametric family that $p\left(\theta|x_{1:n}\right)$ belongs to.

5 The resolving power

In practice, the resolving power of any measurement system is finite. That is, if $\theta_{1}$ and $\theta_{2}$ are too close or similar, one may not expect to detain decisive support for $\theta_{1}$ against $\theta_{2}$ from the data. Often the resolving power differs from one region of the parameter space to another. Taking this into account, in inference, we should give more weight to the regions for which we can potentially detain strong support from the data. As we will demonstrate shortly, this weighting procedure will result in a density, which we shall define as the state of maximum uncertainty.

Although, a priori we do not know the data, nevertheless, the data model density $p\left(x|\theta\right)$ contains some information about the resolving power of the data-generating process. One possible approach is to consider the sensitivity of $p\left(x|\theta\right)$ to the changes in $\theta.$ To this end, the score function (the derivative of the $p\left(x|\theta\right)$ normalized by its value) is a good indication of the sensitivity. It is given by

[TABLE]

We are interested in the score as a function of $\theta.$ For given $X,$ large absolute values of the score indicate high sensitivity and hence high resolving power. Since a priori the data is not known, the true value of the score function is uncertain. Nonetheless, we can consider the mean and the variance of the score as an indication of the resolving power. It can be shown that under some regularity condition, the mean of score function is

[TABLE]

and its variance is

[TABLE]

where $\mathcal{I}\left(\theta\right)$ is also known as the Fisher information. The larger the Fisher information is, the greater is the chance of observing score values at larger distance from zero. Thus, for a given $\theta,$ the large value of the Fisher information indicate high chance of having high resolving power in the neighbourhood of that specific $\theta.$ This observation suggests that the Fisher information can be used for weighting $\theta$ according to its probable degree of resolution. However, at least two problems present themselves:

In general, due to multidimensionality of the parameter $\theta,$ the Fisher information is a matrix and not a scalar. 2. 2.

The dimension of the Fisher information is $\left[\theta^{-2}\right].$ It needs to be $\left[\theta^{-1}\right]$ in order for $m\left(\theta\right)d\theta$ to be dimensionless.

The first problem implies that our resolution might depend on the direction we move. Nevertheless, the volume of the $n$-parallelotope spanned by the column vectors of the Fisher information matrix can be used as an indication of the resolving power. This volume is found by taking the determinant of $\mathcal{I}\left(\theta\right).$ The second problem can be addressed by taking the square root of the volume. Consequently, the density that describes the maximum uncertainty is

[TABLE]

In literature, this density is known as the Jeffreys prior. This result is a direct consequence of our way of assigning prior weights to regions of parameter space with respect to probable resolving power of the likelihood.

6 Induced priors

In the previous sections, we demonstrated that by coarse-graining of average log-likelihood, one can identify the essential information from the distribution of $\theta$ relevant to the centre of mass of the average log-likelihood distribution. This type of information, if attainable, come as a set of conservation laws. We have also argued that Jeffreys prior, eq. (13), can serve as the density with the maximum uncertainty. In the absence of any other information except the functional form of the likelihood, the question about the prior on $\theta$ can be formulated as follows: If we had previously seen $n$ observations, what can we say about the value of $\theta$ before seeing the new observations? In order to be able to answer this question we need the following Lemma and its corollary.

Lemma 1

Let $q(\theta|x)$ dominate $p(\theta|x).$ Then

[TABLE]

Proof.* The result follows from Jensen’s inequality.  * **

Corollary 2

Let $q(\theta|x)$ dominate $p(\theta|x)$ with respect to the common measure $m(\theta).$ Then

[TABLE]

Proof.* The statement follows from lemma 1.  * **

The statement of the above corollary is the same as the Gibbs’ inequality with respect to the common measure $m(\theta).$ Further, it is assumed that $m(\theta)$ is a density which dominates both $q(\theta|x)$ and $p(\theta|x).$

Now, let $f_{k}(\theta)$ be $r$ different functions of $\theta$ such that

[TABLE]

In general, there are many densities which satisfy the above constraints. However, as it will become clear further below, we are interested on the densities with maximum entropy. The following is an extension of the argument given by Jaynes for the discrete case [Jay03, p.357]. Let

[TABLE]

where $Z\left(\lambda_{1},\ldots,\lambda_{r}\right)$ is the normalization constant and $\lambda_{k}$ are functions of $x.$ It can be shown, by the method of Lagrange multipliers, that $q$ satisfies the constraints (15). After substituting $q$ into eq. (14) and taking into account the constraints (15), we get

[TABLE]

This relation holds for all $p$ which are dominated by $q.$ In particular, this inequality holds for the class of all the densities which satisfy the constraints (15) and are dominated by $q$. The equality is only achieved if $p=q.$ In effect, it can be seen that every density with respect to the measure $m$ has lower entropy than $q$. Indeed, notice that all the zeros of $q$ coincide with the zeros of $m.$ Since $m$ dominates all the densities in the class of interest, then we can conclude that $q$ also dominates every density which is dominated by $m$ and hence $q$ dominates every density in the class of interest.

At this point one might raise the following question that why should one pick the density with the maximum entropy. Recall, that the entropy can also be interpreted as a measure of redundancy. Any other family of densities satisfying the constraints (15) have higher redundancy, and hence possibly, larger number of conservation laws, some of which are not induced by the likelihood function.

In the above the numerical values of the observations were irrelevant. Therefore, we can consider them as the pseudo-observations. We are now ready to lay out the algorithm for constructing the likelihood-induced prior. The algorithm is as follows

Determine the average Log-likelihood using the pseudo-observations $x_{1:n}$. 2. 2.

Conduct coarse-graining with respect to $p\left(\theta|x_{1:n}\right)$ and identify the conservation laws. 3. 3.

Determine the Jeffreys’ prior from the data model $p\left(\theta|x\right)$. 4. 4.

Construct the maximum entropy density given by eq. (16)

Let us demonstrate this algorithm for two cases.

6.1 Exponential density

Previously, we described the conservation laws for the data model density being exponential with respect to $x.$ Below we list the results after each step of the algorithm without details.

Average log-likelihood: $l=n^{-1}\ln L=-\ln\theta-\theta^{-1}\overline{\boldsymbol{x}}.$ 2. 2.

Coarse-graining: $E_{\theta}\left(l\right)=-\left\langle\ln\theta\right\rangle-\left\langle\theta^{-1}\right\rangle\overline{\boldsymbol{x}}\Rightarrow$ the conservation laws are $f_{1}\left(\theta\right)=\ln\theta$ and $f_{2}\left(\theta\right)=\theta^{-1}.$ 3. 3.

Jefreys’ prior: $m\left(\theta\right)\propto\theta^{-1}$

Applying the step $4$ of the algorithm results in the parametric family of inverse-gamma distributions

[TABLE]

6.2 Exponential family

A density belongs to the exponential family if it can be expressed in the following way

[TABLE]

where $\boldsymbol{\eta}=$ $\boldsymbol{\eta}_{1:s}\left(\boldsymbol{\theta}\right)$ is the natural parameters, $\boldsymbol{\eta}^{T}\cdot T\left(\boldsymbol{x}\right)=\sum_{i=1}^{s}\eta_{i}T_{i}(\boldsymbol{x})$ and the support of the density does not depend on the choice of $\boldsymbol{\theta}$. If the data model density belongs to the exponential family, then the likelihood of the $n$ iid observations $\boldsymbol{x}_{1:n}$ is

[TABLE]

Below we list the results of each step of the algorithm without details.

Average log-likelihood: $l=n^{-1}\ln L=\overline{\ln h\left(\boldsymbol{x}\right)}+\ln g\left(\boldsymbol{\eta}\right)+\boldsymbol{\eta}^{T}\cdot\overline{T\left(\boldsymbol{x}\right)}$. 2. 2.

Coarse-graining: $E_{\boldsymbol{\theta}}\left(l\right)=\overline{\ln h\left(\boldsymbol{x}\right)}+\left\langle\ln g\left(\boldsymbol{\eta}\right)\right\rangle+\left\langle\boldsymbol{\eta}\right\rangle^{T}\cdot\overline{T\left(\boldsymbol{x}\right)}\Rightarrow$ the conservation laws are $f_{1}\left(\boldsymbol{\theta}\right)=\ln g\left(\boldsymbol{\eta}\right)$ and $f_{2}\left(\boldsymbol{\theta}\right)=\boldsymbol{\eta}.$ 3. 3.

Jeffreys prior: $m\left(\boldsymbol{\theta}\right)\propto\sqrt{\det\mathcal{I}\left(\boldsymbol{\theta}\right)}$

Applying the final step of the algorithm results in

[TABLE]

In literature, the distribution $q$ is known as the conjugate prior for the likelihood $L.$ The conjugate priors play an important role in Bayesian statistics. They are often used because they result in closed form expression for posterior without the need for calculation of the elusive normalization constant. However, the result of this section further elucidates the role of the conjugate priors as not only algebraically convenient constructs but as manifestation of the likelihood-induced information.

7 Concluding remarks

We have demonstrated that, in general, the functional form of the likelihood contains enough information for constructing a prior for the unknown parameters. The key idea was the coarse-graining of the information in order to reduce uncertainty and using entropy as the measure of uncertainty. The identification of the relevant conservation laws along with the measure for resolving power resulted in a four-step algorithm for constructing the likelihood-induced priors. We have also demonstrated that in case the data model density belongs to the exponential family, the likelihood-induced prior is the conjugate prior for the corresponding likelihood. Furthermore, this algorithm can readily be applied to other parametric classes of densities. We shall come back to this issue in the future.