Discovering Behavioral Modes in Deep Reinforcement Learning Policies Using Trajectory Clustering in Latent Space

TL;DR

This paper presents a novel method combining dimensionality reduction and trajectory clustering in latent space to analyze and understand the behavior modes of deep reinforcement learning policies, aiding targeted improvements.

Contribution

It introduces a new approach using PaCMAP and TRACLUS for analyzing DRL policies' latent space, revealing behavior patterns and suboptimal actions.

Findings

Identified diverse behavior patterns in DRL policies

Enhanced policy performance through targeted analysis

Demonstrated effectiveness on Mountain Car task

Abstract

Understanding the behavior of deep reinforcement learning (DRL) agents is crucial for improving their performance and reliability. However, the complexity of their policies often makes them challenging to understand. In this paper, we introduce a new approach for investigating the behavior modes of DRL policies, which involves utilizing dimensionality reduction and trajectory clustering in the latent space of neural networks. Specifically, we use Pairwise Controlled Manifold Approximation Projection (PaCMAP) for dimensionality reduction and TRACLUS for trajectory clustering to analyze the latent space of a DRL policy trained on the Mountain Car control task. Our methodology helps identify diverse behavior patterns and suboptimal choices by the policy, thus allowing for targeted improvements. We demonstrate how our approach, combined with domain knowledge, can enhance a policy's performance in specific regions of the state space.