Policy Entropy for Out-of-Distribution Classification

TL;DR

PEOC is a novel policy entropy-based method for reliably detecting out-of-distribution states in deep reinforcement learning, enhancing safety by identifying unencountered situations.

Contribution

Introduces PEOC, a new out-of-distribution classifier leveraging policy entropy, and provides a structured benchmarking process for RL OOD detection.

Findings

PEOC outperforms existing one-class classifiers in evaluated environments.

Policy entropy effectively signals unencountered states.

Structured benchmarking facilitates comparison of OOD detection methods in RL.

Abstract

One critical prerequisite for the deployment of reinforcement learning systems in the real world is the ability to reliably detect situations on which the agent was not trained. Such situations could lead to potential safety risks when wrong predictions lead to the execution of harmful actions. In this work, we propose PEOC, a new policy entropy based out-of-distribution classifier that reliably detects unencountered states in deep reinforcement learning. It is based on using the entropy of an agent's policy as the classification score of a one-class classifier. We evaluate our approach using a procedural environment generator. Results show that PEOC is highly competitive against state-of-the-art one-class classification algorithms on the evaluated environments. Furthermore, we present a structured process for benchmarking out-of-distribution classification in reinforcement learning.