Criticality and Safety Margins for Reinforcement Learning

TL;DR

This paper introduces a framework for quantifying the criticality of decisions in reinforcement learning, enabling safer deployment by identifying potentially unsafe situations through interpretable safety margins and ground-truth metrics.

Contribution

It defines true and proxy criticality metrics with a clear significance to users, and demonstrates their effectiveness in predicting unsafe situations in RL agents.

Findings

Proxy criticality correlates monotonically with true criticality.

Safety margins can identify high-risk decision points.

Supervising 5% of decisions could prevent nearly half of agent errors.

Abstract

State of the art reinforcement learning methods sometimes encounter unsafe situations. Identifying when these situations occur is of interest both for post-hoc analysis and during deployment, where it might be advantageous to call out to a human overseer for help. Efforts to gauge the criticality of different points in time have been developed, but their accuracy is not well established due to a lack of ground truth, and they are not designed to be easily interpretable by end users. Therefore, we seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users. We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions. We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality. Safety margins make these interpretable, when defined as the number of random actions for which performance loss will not exceed some tolerance with high confidence. We demonstrate this approach in several environment-agent combinations; for an A3C agent in an Atari Beamrider environment, the lowest 5% of safety margins contain 47% of agent losses; i.e., supervising only 5% of decisions could potentially prevent roughly half of an agent's errors. This criticality framework measures the potential impacts of bad decisions, even before those decisions are made, allowing for more effective debugging and oversight of autonomous agents.