DRLViz: Understanding Decisions and Memory in Deep Reinforcement Learning

TL;DR

DRLViz is a visual analytics tool designed to interpret the complex internal memory of deep reinforcement learning agents, aiding experts in understanding decision-making processes and improving model transparency.

Contribution

This paper introduces DRLViz, a novel visual interface that helps interpret high-dimensional memory in deep reinforcement learning agents, enhancing explainability and analysis capabilities.

Findings

DRLViz enables effective interpretation of agent memory in navigation tasks.

Experts find DRLViz useful for diagnosing errors and understanding decision factors.

The approach is applicable beyond simulated environments to real-world scenarios.

Abstract

We present DRLViz, a visual analytics interface to interpret the internal memory of an agent (e.g. a robot) trained using deep reinforcement learning. This memory is composed of large temporal vectors updated when the agent moves in an environment and is not trivial to understand due to the number of dimensions, dependencies to past vectors, spatial/temporal correlations, and co-correlation between dimensions. It is often referred to as a black box as only inputs (images) and outputs (actions) are intelligible for humans. Using DRLViz, experts are assisted to interpret decisions using memory reduction interactions, and to investigate the role of parts of the memory when errors have been made (e.g. wrong direction). We report on DRLViz applied in the context of video games simulators (ViZDoom) for a navigation scenario with item gathering tasks. We also report on experts evaluation using DRLViz, and applicability of DRLViz to other scenarios and navigation problems beyond simulation games, as well as its contribution to black box models interpretability and explainability in the field of visual analytics.