BXRL: Behavior-Explainable Reinforcement Learning

TL;DR

BXRL introduces a formal framework for explaining behaviors in reinforcement learning by defining behaviors as measurable patterns of actions, enabling more precise and interpretable explanations of agent actions and policies.

Contribution

The paper formalizes behavior in reinforcement learning, introduces a behavior measure, and analyzes how existing explainability methods can be adapted to explain behaviors.

Findings

Defined behavior as a function from policies to real numbers.

Proposed contrastive behaviors for explanation.

Ported HighwayEnv to JAX for behavior analysis.

Abstract

A major challenge of Reinforcement Learning is that agents often learn undesired behaviors that seem to defy the reward structure they were given. Explainable Reinforcement Learning (XRL) methods can answer queries such as "explain this specific action", "explain this specific trajectory", and "explain the entire policy". However, XRL lacks a formal definition for behavior as a pattern of actions across many episodes. We provide such a definition, and use it to enable a new query: "Explain this behavior". We present Behavior-Explainable Reinforcement Learning (BXRL), a new problem formulation that treats behaviors as first-class objects. BXRL defines a behavior measure as any function $m : \Pi \to \mathbb{R}$, allowing users to precisely express the pattern of actions that they find interesting and measure how strongly the policy exhibits it. We define contrastive behaviors that reduce the question "why does the agent prefer $a$ to $a'$?" to "why is $m(\pi)$ high?" which can be explored with differentiation. We do not implement an explainability method; we instead analyze three existing methods and propose how they could be adapted to explain behavior. We present a port of the HighwayEnv driving environment to JAX, which provides an interface for defining, measuring, and differentiating behaviors with respect to the model parameters.