Re-understanding Finite-State Representations of Recurrent Policy Networks

TL;DR

This paper presents a new method for analyzing recurrent neural network policies by using unminimized finite-state machines and interpretability tools, revealing novel insights into their decision-making processes.

Contribution

It introduces an analysis approach that preserves key decision points in FSMs and an attention tool for understanding observation roles, improving interpretability of recurrent policies.

Findings

Revealed new insights into policy behaviors in Atari games.

Demonstrated the effectiveness of unminimized FSM analysis.

Provided tools for deeper understanding of observation influence.

Abstract

We introduce an approach for understanding control policies represented as recurrent neural networks. Recent work has approached this problem by transforming such recurrent policy networks into finite-state machines (FSM) and then analyzing the equivalent minimized FSM. While this led to interesting insights, the minimization process can obscure a deeper understanding of a machine's operation by merging states that are semantically distinct. To address this issue, we introduce an analysis approach that starts with an unminimized FSM and applies more-interpretable reductions that preserve the key decision points of the policy. We also contribute an attention tool to attain a deeper understanding of the role of observations in the decisions. Our case studies on 7 Atari games and 3 control benchmarks demonstrate that the approach can reveal insights that have not been previously noticed.