Interpretable Learning Dynamics in Unsupervised Reinforcement Learning

TL;DR

This paper introduces an interpretability framework for unsupervised reinforcement learning agents, analyzing how intrinsic motivation influences attention, behavior, and internal representations, with new metrics for attention dynamics.

Contribution

It proposes a novel interpretability framework and metrics to analyze internal dynamics of unsupervised RL agents, highlighting the effects of curiosity-driven exploration and architecture.

Findings

Curiosity-driven agents exhibit broader, more dynamic attention.

Transformer-RND combines wide attention with structured representations.

Architectural biases and training signals significantly influence agent behavior.

Abstract

We present an interpretability framework for unsupervised reinforcement learning (URL) agents, aimed at understanding how intrinsic motivation shapes attention, behavior, and representation learning. We analyze five agents DQN, RND, ICM, PPO, and a Transformer-RND variant trained on procedurally generated environments, using Grad-CAM, Layer-wise Relevance Propagation (LRP), exploration metrics, and latent space clustering. To capture how agents perceive and adapt over time, we introduce two metrics: attention diversity, which measures the spatial breadth of focus, and attention change rate, which quantifies temporal shifts in attention. Our findings show that curiosity-driven agents display broader, more dynamic attention and exploratory behavior than their extrinsically motivated counterparts. Among them, TransformerRND combines wide attention, high exploration coverage, and compact, structured latent representations. Our results highlight the influence of architectural inductive biases and training signals on internal agent dynamics. Beyond reward-centric evaluation, the proposed framework offers diagnostic tools to probe perception and abstraction in RL agents, enabling more interpretable and generalizable behavior.