STACHE: Local Black-Box Explanations for Reinforcement Learning Policies

TL;DR

STACHE is a framework that provides local, black-box explanations for reinforcement learning policies by identifying invariant action regions and minimal state changes needed to alter decisions, aiding debugging and understanding.

Contribution

It introduces a novel exact, search-based method exploiting factored state spaces to generate local explanations without surrogate models, improving interpretability of RL policies.

Findings

Effectively explains policy actions in Gymnasium environments.

Captures policy evolution from unstable to robust strategies.

Provides actionable insights into agent sensitivity and decision boundaries.

Abstract

Reinforcement learning agents often behave unexpectedly in sparse-reward or safety-critical environments, creating a strong need for reliable debugging and verification tools. In this paper, we propose STACHE, a comprehensive framework for generating local, black-box explanations for an agent's specific action within discrete Markov games. Our method produces a Composite Explanation consisting of two complementary components: (1) a Robustness Region, the connected neighborhood of states where the agent's action remains invariant, and (2) Minimal Counterfactuals, the smallest state perturbations required to alter that decision. By exploiting the structure of factored state spaces, we introduce an exact, search-based algorithm that circumvents the fidelity gaps of surrogate models. Empirical validation on Gymnasium environments demonstrates that our framework not only explains policy actions, but also effectively captures the evolution of policy logic during training - from erratic, unstable behavior to optimized, robust strategies - providing actionable insights into agent sensitivity and decision boundaries.