VRAIL: Vectorized Reward-based Attribution for Interpretable Learning

TL;DR

VRAIL introduces a bi-level, interpretable framework for value-based reinforcement learning that improves training stability and reveals meaningful subgoals through feature attribution.

Contribution

VRAIL presents a novel, model-agnostic approach combining value function estimation with reward shaping for enhanced interpretability in RL.

Findings

Improves training stability and convergence over standard DQN.

Uncovers human-interpretable subgoals like passenger possession.

Demonstrates effectiveness in Taxi-v3 environment.

Abstract

We propose VRAIL (Vectorized Reward-based Attribution for Interpretable Learning), a bi-level framework for value-based reinforcement learning (RL) that learns interpretable weight representations from state features. VRAIL consists of two stages: a deep learning (DL) stage that fits an estimated value function using state features, and an RL stage that uses this to shape learning via potential-based reward transformations. The estimator is modeled in either linear or quadratic form, allowing attribution of importance to individual features and their interactions. Empirical results on the Taxi-v3 environment demonstrate that VRAIL improves training stability and convergence compared to standard DQN, without requiring environment modifications. Further analysis shows that VRAIL uncovers semantically meaningful subgoals, such as passenger possession, highlighting its ability to produce human-interpretable behavior. Our findings suggest that VRAIL serves as a general, model-agnostic framework for reward shaping that enhances both learning and interpretability.