Functional Estimation of the Marginal Likelihood

TL;DR

This paper introduces a flexible framework for estimating and optimizing the marginal likelihood in complex statistical models, enabling comprehensive inference across hyperparameters using posterior samples and establishing estimator consistency.

Contribution

It presents a novel method that unifies various existing techniques for marginal likelihood estimation and demonstrates its consistency and applicability to complex models.

Findings

Method relates to existing techniques like SMC and Gibbs sampling

Establishes estimator consistency with increasing sampling effort

Successfully applied to Gaussian process models and crossed effect models

Abstract

We propose a framework for computing, optimizing and integrating with respect to a smooth marginal likelihood in statistical models that involve high-dimensional parameters/latent variables and continuous low-dimensional hyperparameters. The method requires samples from the posterior distribution of the parameters for different values of the hyperparameters on a simulation grid and returns inference on the marginal likelihood defined everywhere on its domain, and on its functionals. We show how the method relates to many of the methods that have been used in this context, including sequential Monte Carlo, Gibbs sampling, Monte Carlo maximum likelihood, and umbrella sampling. We establish the consistency of the proposed estimators as the sampling effort increases, both when the simulation grid is kept fixed and when it becomes dense in the domain. We showcase the approach on Gaussian process regression and classification and crossed effect models.