General bounds on the quality of Bayesian coresets

TL;DR

This paper establishes general theoretical bounds on the approximation error of Bayesian coresets, broadening understanding of their limitations and performance across diverse models, including complex and heavy-tailed distributions.

Contribution

It provides the first general upper and lower bounds on coreset approximation quality applicable to a wide range of Bayesian models, beyond restrictive assumptions.

Findings

Lower bounds reveal fundamental limitations of coreset approximations.

Upper bounds analyze performance of recent subsample-optimize methods.

Experimental validation on complex Bayesian posteriors demonstrates theory's applicability.

Abstract

Bayesian coresets speed up posterior inference in the large-scale data regime by approximating the full-data log-likelihood function with a surrogate log-likelihood based on a small, weighted subset of the data. But while Bayesian coresets and methods for construction are applicable in a wide range of models, existing theoretical analysis of the posterior inferential error incurred by coreset approximations only apply in restrictive settings -- i.e., exponential family models, or models with strong log-concavity and smoothness assumptions. This work presents general upper and lower bounds on the Kullback-Leibler (KL) divergence of coreset approximations that reflect the full range of applicability of Bayesian coresets. The lower bounds require only mild model assumptions typical of Bayesian asymptotic analyses, while the upper bounds require the log-likelihood functions to satisfy a generalized subexponentiality criterion that is weaker than conditions used in earlier work. The lower bounds are applied to obtain fundamental limitations on the quality of coreset approximations, and to provide a theoretical explanation for the previously-observed poor empirical performance of importance sampling-based construction methods. The upper bounds are used to analyze the performance of recent subsample-optimize methods. The flexibility of the theory is demonstrated in validation experiments involving multimodal, unidentifiable, heavy-tailed Bayesian posterior distributions.