General adjoint-differentiated Laplace approximation

TL;DR

This paper introduces a generalized adjoint-differentiated Laplace approximation for hierarchical models, enabling efficient gradient-based inference with broader likelihood applicability without additional computational costs.

Contribution

It extends the adjoint-differentiated Laplace approximation to handle a wider range of likelihoods without requiring analytical derivatives, improving flexibility and efficiency.

Findings

The generalized method is slightly faster than existing approaches on standard LGMs.

It successfully applies to LGMs with unconventional likelihoods.

The approach maintains computational efficiency despite increased flexibility.

Abstract

The hierarchical prior used in Latent Gaussian models (LGMs) induces a posterior geometry prone to frustrate inference algorithms. Marginalizing out the latent Gaussian variable using an integrated Laplace approximation removes the offending geometry, allowing us to do efficient inference on the hyperparameters. To use gradient-based inference we need to compute the approximate marginal likelihood and its gradient. The adjoint-differentiated Laplace approximation differentiates the marginal likelihood and scales well with the dimension of the hyperparameters. While this method can be applied to LGMs with any prior covariance, it only works for likelihoods with a diagonal Hessian. Furthermore, the algorithm requires methods which compute the first three derivatives of the likelihood with current implementations relying on analytical derivatives. I propose a generalization which is applicable to a broader class of likelihoods and does not require analytical derivatives of the likelihood. Numerical experiments suggest the added flexibility comes at no computational cost: on a standard LGM, the new method is in fact slightly faster than the existing adjoint-differentiated Laplace approximation. I also apply the general method to an LGM with an unconventional likelihood. This example highlights the algorithm's potential, as well as persistent challenges.