Scalable Safety-Critical Policy Evaluation with Accelerated Rare Event Sampling

TL;DR

This paper introduces the Accelerated Policy Evaluation (APE) method for efficiently estimating rare event probabilities in reinforcement learning, improving scalability and accuracy in safety-critical systems.

Contribution

The paper presents APE, a novel approach that uses adaptive importance sampling and adversarial environment modeling to efficiently evaluate rare events in large or continuous spaces.

Findings

APE estimates rare event probabilities with less bias.

APE requires significantly fewer samples than baseline methods.

Empirical results demonstrate APE's scalability and effectiveness.

Abstract

Evaluating rare but high-stakes events is one of the main challenges in obtaining reliable reinforcement learning policies, especially in large or infinite state/action spaces where limited scalability dictates a prohibitively large number of testing iterations. On the other hand, a biased or inaccurate policy evaluation in a safety-critical system could potentially cause unexpected catastrophic failures during deployment. This paper proposes the Accelerated Policy Evaluation (APE) method, which simultaneously uncovers rare events and estimates the rare event probability in Markov decision processes. The APE method treats the environment nature as an adversarial agent and learns towards, through adaptive importance sampling, the zero-variance sampling distribution for the policy evaluation. Moreover, APE is scalable to large discrete or continuous spaces by incorporating function approximators. We investigate the convergence property of APE in the tabular setting. Our empirical studies show that APE can estimate the rare event probability with a smaller bias while only using orders of magnitude fewer samples than baselines in multi-agent and single-agent environments.