Rethinking the Effective Sample Size

TL;DR

This paper critically examines the commonly used approximation of the effective sample size in importance sampling, revealing its limitations and proposing directions for better metrics.

Contribution

It challenges the validity of the widely used ESS approximation in importance sampling and discusses potential improvements and alternatives.

Findings

The standard ESS approximation is based on questionable assumptions.

Numerical examples demonstrate the limitations of the current ESS approximation.

The paper suggests exploring alternative metrics for assessing sample quality.

Abstract

The effective sample size (ESS) is widely used in sample-based simulation methods for assessing the quality of a Monte Carlo approximation of a given distribution and of related integrals. In this paper, we revisit the approximation of the ESS in the specific context of importance sampling (IS). The derivation of this approximation, that we will denote as $\widehat{\text{ESS}}$, is partially available in Kong (1992). This approximation has been widely used in the last 25 years due to its simplicity as a practical rule of thumb in a wide variety of importance sampling methods. However, we show that the multiple assumptions and approximations in the derivation of $\widehat{\text{ESS}}$, makes it difficult to be considered even as a reasonable approximation of the ESS. We extend the discussion of the $\widehat{\text{ESS}}$ in the multiple importance sampling (MIS) setting, we display numerical examples, and we discuss several avenues for developing alternative metrics. This paper does not cover the use of ESS for MCMC algorithms.