Understanding and mitigating difficulties in posterior predictive evaluation

TL;DR

This paper investigates the low signal-to-noise ratio problem in posterior predictive density estimation and proposes an importance sampling method with test-time optimization to improve accuracy.

Contribution

It identifies the exponential decay of SNR in posterior predictive evaluation and introduces an importance sampling approach with test-time optimization to mitigate this issue.

Findings

SNR decays exponentially with data mismatch, high latent dimensions, or large test sets.

Importance sampling with test-time optimized proposals significantly improves estimation accuracy.

The proposed method enhances posterior predictive evaluation in approximate Bayesian inference.

Abstract

Predictive posterior densities (PPDs) are of interest in approximate Bayesian inference. Typically, these are estimated by simple Monte Carlo (MC) averages using samples from the approximate posterior. We observe that the signal-to-noise ratio (SNR) of such estimators can be extremely low. An analysis for exact inference reveals SNR decays exponentially as there is an increase in (a) the mismatch between training and test data, (b) the dimensionality of the latent space, or (c) the size of the test data relative to the training data. Further analysis extends these results to approximate inference. To remedy the low SNR problem, we propose replacing simple MC sampling with importance sampling using a proposal distribution optimized at test time on a variational proxy for the SNR and demonstrate that this yields greatly improved estimates.