Optimal Particle-based Approximation of Discrete Distributions (OPAD)

TL;DR

This paper proves that for discrete target distributions, there exists a unique optimal weighting of particles that minimizes KL divergence, and shows how existing particle methods can be improved with minimal modifications.

Contribution

It introduces a theoretically optimal weighting scheme for particle-based methods targeting discrete distributions, enhancing their accuracy without extra computational costs.

Findings

Optimal weights minimize KL divergence for discrete targets.

Reweighting improves particle approximation accuracy.

Empirical results show substantial performance gains.

Abstract

Particle-based methods include a variety of techniques, such as Markov Chain Monte Carlo (MCMC) and Sequential Monte Carlo (SMC), for approximating a probabilistic target distribution with a set of weighted particles. In this paper, we prove that for any set of particles, there is a unique weighting mechanism that minimizes the Kullback-Leibler (KL) divergence of the (particle-based) approximation from the target distribution, when that distribution is discrete -- any other weighting mechanism (e.g. MCMC weighting that is based on particles' repetitions in the Markov chain) is sub-optimal with respect to this divergence measure. Our proof does not require any restrictions either on the target distribution, or the process by which the particles are generated, other than the discreteness of the target. We show that the optimal weights can be determined based on values that any existing particle-based method already computes; As such, with minimal modifications and no extra computational costs, the performance of any particle-based method can be improved. Our empirical evaluations are carried out on important applications of discrete distributions including Bayesian Variable Selection and Bayesian Structure Learning. The results illustrate that our proposed reweighting of the particles improves any particle-based approximation to the target distribution consistently and often substantially.