Learning Finite State Representations of Recurrent Policy Networks

TL;DR

This paper introduces Quantized Bottleneck Insertion to create finite, interpretable representations of RNN policies, enabling better understanding and analysis of their memory and behavior in reinforcement learning tasks.

Contribution

The paper presents a novel method for quantizing RNN memory and features, producing small, interpretable finite state representations of policies.

Findings

Finite representations as small as 3 states and 10 observations for Pong.

Quantized representations improve interpretability of RNN policies.

Effective on synthetic environments and Atari games.

Abstract

Recurrent neural networks (RNNs) are an effective representation of control policies for a wide range of reinforcement and imitation learning problems. RNN policies, however, are particularly difficult to explain, understand, and analyze due to their use of continuous-valued memory vectors and observation features. In this paper, we introduce a new technique, Quantized Bottleneck Insertion, to learn finite representations of these vectors and features. The result is a quantized representation of the RNN that can be analyzed to improve our understanding of memory use and general behavior. We present results of this approach on synthetic environments and six Atari games. The resulting finite representations are surprisingly small in some cases, using as few as 3 discrete memory states and 10 observations for a perfect Pong policy. We also show that these finite policy representations lead to improved interpretability.