SHAP-Guided Kernel Actor-Critic for Explainable Reinforcement Learning

TL;DR

This paper introduces RSA2C, an explainable reinforcement learning algorithm that uses SHAP-based attributions within a kernelized actor-critic framework to improve interpretability, stability, and efficiency.

Contribution

It proposes a novel attribution-aware, kernelized actor-critic method utilizing SHAP for state attribution, enhancing interpretability and stability in RL.

Findings

RSA2C outperforms baselines in continuous-control tasks.

It provides stable convergence bounds under state perturbations.

RSA2C offers improved interpretability through state attributions.

Abstract

Actor-critic (AC) methods are a cornerstone of reinforcement learning (RL) but offer limited interpretability. Current explainable RL methods seldom use state attributions to assist training. Rather, they treat all state features equally, thereby neglecting the heterogeneous impacts of individual state dimensions on the reward. We propose RKHS-SHAP-based Advanced Actor-Critic (RSA2C), an attribution-aware, kernelized, two-timescale AC algorithm, including Actor, Value Critic, and Advantage Critic. The Actor is instantiated in a vector-valued reproducing kernel Hilbert space (RKHS) with a Mahalanobis-weighted operator-valued kernel, while the Value Critic and Advantage Critic reside in scalar RKHSs. These RKHS-enhanced components use sparsified dictionaries: the Value Critic maintains its own dictionary, while the Actor and Advantage Critic share one. State attributions, computed from the Value Critic via RKHS-SHAP (kernel mean embedding for on-manifold and conditional mean embedding for off-manifold expectations), are converted into Mahalanobis-gated weights that modulate Actor gradients and Advantage Critic targets. We derive a global, non-asymptotic convergence bound under state perturbations, showing stability through the perturbation-error term and efficiency through the convergence-error term. Empirical results on three continuous-control environments show that RSA2C achieves efficiency, stability, and interpretability.