Are profile likelihoods likelihoods? No, but sometimes they can be

TL;DR

This paper explores the relationship between profile likelihoods and marginal likelihoods, demonstrating their equivalence in normal models using the Jeffreys prior, and discusses the uniqueness of this case.

Contribution

It shows that profile likelihoods can be true likelihoods in normal models through their equivalence to marginal likelihoods with Jeffreys prior.

Findings

Profile likelihoods can be identical to marginal likelihoods in normal models.

The Jeffreys prior achieves this equivalence.

Such an equivalence is likely unique to normal models.

Abstract

We offer our two cents to the ongoing discussion on whether profile likelihoods are "true" likelihood functions, by showing that the profile likelihood function can in fact be identical to a marginal likelihood in the special case of normal models. Thus, profile likelihoods can be "true" likelihoods insofar as marginal likelihoods are "true" likelihoods. The prior distribution that achieves this equivalence turns out to be the Jeffreys prior. We suspect, however, that normal models are the only class of models for which such an equivalence between maximization and marginalization is exact.

Figures

Equations

$$p(y|\mu ,{\sigma }^2)\propto ({\sigma }^2{)}^{-n/2}\exp \left\{-\frac{1}{2{\sigma }^2}\sum _{i=1}^n(y_i-\mu {)}^2\right\}.$$

$${\hat{\sigma }}^2(\mu )=\frac{1}{n}\sum _{i=1}^n(y_i-\mu {)}^2,$$

$$\underset{{\sigma }^2}{\sup }p(y|\mu ,{\sigma }^2)=p(y|\mu ,{\hat{\sigma }}^2(\mu ))\propto {\left[\sum _{i=1}^n(y_i-\mu {)}^2\right]}^{-n/2}.$$

$$p(y|\mu )$$

$$\underset{{\sigma }^2}{\sup }p(y|\mu ,{\sigma }^2)\equiv {\int }_{{\sigma }^2}p(y|\mu ,{\sigma }^2)p({\sigma }^2)d({\sigma }^2)$$

$$\underset{{\sigma }^2}{\sup }p(y|\mu ,{\sigma }^2)p(\mu )\propto {\int }_{{\sigma }^2}p(y|\mu ,{\sigma }^2)p(\mu )p({\sigma }^2)d({\sigma }^2),$$

Full text

profile likelihood,

marginal likelihood,

true likelihood,

equivalent prior,

Jeffreys prior,

, and

1 Introduction

There has been some recent and rather lively debate as to whether the profile likelihood, obtained by maximizing out nuisance parameters in the full likelihood, can be considered a “true” likelihood function in the remaining parameters, with arguments ranging from probabilistic, possibilistic and even philosophical perspectives (e.g., Aitkin, 2005, 2010; Evans, 2015; Maclaren, 2018; Robert, 2018). Here, the notion of a “true” likelihood is a function that corresponds to some joint probability distribution on the data for each value of the model parameters111Maclaren (2018) argues for a different notion of likelihood based on possibility rather than probability, with addition replaced by maximization. This approach is worth further consideration, however in this note we stick with the classical probability-based notion of likelihood..

The consensus from the statistical literature seems to be “no”, in general. Aitkin (2005) states rather unequivocally that “the profile likelihood is not a likelihood, but a likelihood maximized over nuisance parameters given the values of the parameters of interest.” In other words, the maximization operator does not generally take probability distributions to probability distributions, but merely to a “slice” in a probability distribution (hence the “profile” moniker).

Of course, from a frequentist point of view the profile likelihood can still exhibit likelihood-type statistical properties, regardless of whether or not it corresponds to a true likelihood. These properties include consistency, asymptotic normality and asymptotic efficiency of its maximizer, with the profile likelihood ratio test even exhibiting Wilks’ phenomenon under some general conditions (Murphy & van der Vaart, 2000).

From a Bayesian point of view, nuisance parameters are usually dealt with via marginalization instead of maximization. In contrast to the profile likelihood, there is little debate as to whether the marginal likelihood corresponds to a true likelihood, as “integration over variables takes probability distributions to probability distributions” (Maclaren, 2018). Indeed, the elementary concept of a marginal probability is constructed precisely by integrating joint probabilities over a subset of variables.

While maximization and marginalization are two seemingly disparate operators, it turns out that in the special case of normal models, the profile likelihood for the mean parameter(s) is precisely equivalent to the marginal likelihood obtained by integrating over Jeffreys prior on the nuisance variance parameters. In this case, profile likelihood can be considered a true likelihood insofar as a marginal likelihood is a true likelihood. This equivalence is exact for normal models, and we speculate that, like other results from likelihood theory, it may only be asymptotically true for other exponential families.

2 Profile likelihood, marginal likelihood and the equivalent prior for normal models

Let $y=(y_{1},y_{2},\ldots,y_{n})^{\top}$ be a random sample from a normal distribution with mean $\mu$ and variance $\sigma^{2}$. The likelihood function (using Bayesian notation) is given by

[TABLE]

The maximum likelihood estimator of $\sigma^{2}$ for each given $\mu$ is

[TABLE]

so that the profile likelihood for $\mu$ is

[TABLE]

It is not immediately clear that this function corresponds to a valid probability distribution in $y$ for each $\mu$. This kind of ambiguity is precisely what has fuelled the debate over whether profile likelihoods can be considered true likelihoods.

On the other hand, consider a Jeffreys prior $p(\sigma^{2})\propto 1/\sigma^{2}$ on the variance $\sigma^{2}$ where the mean $\mu$ is treated as given. Integrating out $\sigma^{2}$ leads to the marginal likelihood as

[TABLE]

by noticing that the integrand is the kernel of a Inverse-Gamma distribution with shape parameter $n/2$ and scale parameter $\sum_{i=1}^{n}(y_{i}-\mu)^{2}/2$. We see that the marginal likelihood coincides exactly with the profile likelihood, that is,

[TABLE]

for Jeffreys prior $p(\sigma^{2})\propto 1/\sigma^{2}$ on the variance $\sigma^{2}$.

A practical consequence is that the profile likelihood can be used to construct valid posterior distributions for Bayesian inferences. Given a prior $p(\mu)$ on $\mu$, the ‘profile posterior” is

[TABLE]

precisely the same as the marginal posterior obtained from integrating over Jeffreys prior on $\sigma^{2}$. For example, profile posterior distributions resulting from priors $p(\mu)\sim N(0,1^{2}),p(\mu)\sim N(0,2^{2}),$ and the improper prior $p(\mu)\propto 1$ can be obtained via Gibbs sampling, following Chapter 8.2.1 of Kroese & Chan (2014), say. Indeed, following the example in that book, Figure 1 displays the corresponding profile posterior distributions for $\mu$ given a random sample of 10 observations from a standard normal distribution.

The Jeffreys prior can therefore be thought of as an “equivalent prior” that makes marginalizing the likelihood equivalent to maximizing the likelihood. Analogous results for the multivariate and regression cases are also elementary to show.

Result 2.1**.**

Let $y_{1},y_{2},\ldots,y_{n}\stackrel{{\scriptstyle\rm{iid}}}{{\sim}}N_{d}(\mu,\Sigma)$, where $\mu\in\mathbf{R}^{d}$ is a $d$-vector of mean parameters of interest and $\Sigma$ is a $d\times d$ nuisance variance matrix. Then the profile likelihood for $\mu$ is equivalent to the marginal likelihood for $\mu$ for Jeffreys prior $p(\Sigma)\propto|\Sigma|^{-(d+1)/2}$ on $\Sigma$.

Result 2.2**.**

Let $y_{i}|x_{i}\stackrel{{\scriptstyle\rm ind}}{{\sim}}N(x_{i}^{\top}\beta,\sigma^{2})$, $i=1,2,\ldots,n$, where each $x_{i}\in\mathbf{R}^{q}$ is a vector of covariates, $\beta$ is an associated vector of mean parameters of interest and $\sigma^{2}$ is a nuisance variance parameter. Then the profile likelihood for $\beta$ is equivalent to the marginal likelihood for $\beta$ for Jeffreys prior $p(\sigma^{2})\propto 1/\sigma^{2}$ on $\sigma^{2}$.

3 Discussion

Our contribution to the ongoing debate over the nature of the profile likelihood is to provide a simple (counter-)example in which the profile likelihood is identical to a marginal likelihood. We find it rather remarkable and somewhat counter-intuitive that marginalization can be made equivalent to maximization via a particular choice of prior on the nuisance parameters. That this equivalent prior happens to be the well-known Jeffreys prior is also an interesting coincidence, but perhaps not completely unexpected as both the profile likelihood and Jeffreys prior are constructed to be “non-informative” in some frequentist or Bayesian sense, respectively. Of course, whether the improper Jeffreys prior constitutes a “true” prior that can be integrated over is another debate, perhaps for another day.

In keeping with Aitkin (2005), we suspect that normal models are the only class of models for which this equivalence is exact. However, we also speculate that a generalization may hold asymptotically for other exponential families. Heuristically speaking, the score equations for exponential families, whilst typically not solvable in closed-form, can be linearized in its parameters, with leading term proportional to the Hessian of the likelihood, the inverse of which forms the basis of Jeffreys prior. It is also well-known that exponential families for data induce exponential families in the model parameters (which is why exponential families always have conjugate priors). These two ingredients combine to give us hope that the (linearized) profile likelihood might pop up as the normalizing constant when integrating out an exponential family likelihood over the Jeffreys prior, just as it did in the normal case. This is a lead worth exploring further.

Acknowledgements

We thank Dr Yao-ban Chan (Melbourne) for comments that improved this note.