Over-Conservativeness of Variance-Based Efficiency Criteria and Probabilistic Efficiency in Rare-Event Simulation

TL;DR

This paper challenges traditional variance-based efficiency criteria in rare-event simulation, proposing a probabilistic efficiency concept that allows for effective importance sampling using only the most significant dominating points, especially in high dimensions.

Contribution

It introduces a new probabilistic efficiency notion and demonstrates that focusing on the most significant dominating points suffices under large deviations regimes, reducing computational complexity.

Findings

Traditional criteria may be loose and overly conservative.

Using only the most significant dominating points can achieve probabilistic efficiency.

This approach is particularly effective in high-dimensional problems.

Abstract

In rare-event simulation, an importance sampling (IS) estimator is regarded as efficient if its relative error, namely the ratio between its standard deviation and mean, is sufficiently controlled. It is widely known that when a rare-event set contains multiple "important regions" encoded by the so-called dominating points, IS needs to account for all of them via mixing to achieve efficiency. We argue that in typical experiments, missing less significant dominating points may not necessarily cause inefficiency, and the traditional analysis recipe could suffer from intrinsic looseness by using relative error, or in turn estimation variance, as an efficiency criterion. We propose a new efficiency notion, which we call probabilistic efficiency, to tighten this gap. In particular, we show that under the standard Gartner-Ellis large deviations regime, an IS that uses only the most significant dominating points is sufficient to attain this efficiency notion. Our finding is especially relevant in high-dimensional settings where the computational effort to locate all dominating points is enormous.