Toward Explainable Offline RL: Analyzing Representations in Intrinsically Motivated Decision Transformers

TL;DR

This paper introduces a post-hoc explainability framework for Elastic Decision Transformers in offline reinforcement learning, revealing how intrinsic motivation influences learned representations and improves policy performance.

Contribution

It provides a systematic analysis of how intrinsic motivation mechanisms shape embedding structures in EDTs, uncovering environment-specific representational patterns.

Findings

Intrinsic motivation creates distinct embedding structures.

Embedding metrics correlate with performance in environment-specific ways.

Intrinsic motivation acts as a representational prior shaping decision-making.

Abstract

Elastic Decision Transformers (EDTs) have proved to be particularly successful in offline reinforcement learning, offering a flexible framework that unifies sequence modeling with decision-making under uncertainty. Recent research has shown that incorporating intrinsic motivation mechanisms into EDTs improves performance across exploration tasks, yet the representational mechanisms underlying these improvements remain unexplored. In this paper, we introduce a systematic post-hoc explainability framework to analyze how intrinsic motivation shapes learned embeddings in EDTs. Through statistical analysis of embedding properties (including covariance structure, vector magnitudes, and orthogonality), we reveal that different intrinsic motivation variants create fundamentally different representational structures. Our analysis demonstrates environment-specific correlation patterns between embedding metrics and performance that explain why intrinsic motivation improves policy learning. These findings show that intrinsic motivation operates beyond simple exploration bonuses, acting as a representational prior that shapes embedding geometry in biologically plausible ways, creating environment-specific organizational structures that facilitate better decision-making.