Full version: An evaluation of estimation techniques for probabilistic reachability

TL;DR

This paper evaluates Monte Carlo, Quasi-Monte Carlo, and Randomised Quasi-Monte Carlo methods for probabilistic reachability in hybrid systems, focusing on error estimation and confidence interval techniques.

Contribution

It introduces a new CLT-based approach for confidence intervals near probability borders and compares various interval estimation methods.

Findings

QMC methods show faster convergence than MC.

The new CLT-based approach provides reliable confidence intervals near 0 and 1.

Guidelines for probability estimation techniques are proposed.

Abstract

We evaluate numerically-precise Monte Carlo (MC), Quasi-Monte Carlo (QMC) and Randomised Quasi-Monte Carlo (RQMC) methods for computing probabilistic reachability in hybrid systems with random parameters. Computing reachability probability amounts to computing (multidimensional) integrals. In particular, we pay attention to QMC methods due to their theoretical benefits in convergence speed with respect to the MC method. The Koksma-Hlawka inequality is a standard result that bounds the approximation of an integral by QMC techniques. However, it is not useful in practice because it depends on the variation of the integrand function, which is in general difficult to compute. The question arises whether it is possible to apply statistical or empirical methods for estimating the approximation error. In this paper we compare a number of interval estimation techniques based on the Central Limit Theorem (CLT), and we also introduce a new approach based on the CLT for computing confidence intervals for probability near the borders of the [0,1] interval. Based on our analysis, we provide justification for the use of the developed approach and suggest usage guidelines for probability estimation techniques.