Learning of Generalizable and Interpretable Knowledge in Grid-Based Reinforcement Learning Environments

TL;DR

This paper introduces a method using program synthesis to interpret and understand deep reinforcement learning agents in grid-based environments, enhancing interpretability and verification of learned behaviors.

Contribution

It adapts DreamCoder for learning interpretable concepts in grid environments, enabling analysis of agent behavior through generated programs and decision visualization.

Findings

Programs reveal learned concepts and behaviors.

Visualization aids understanding of agent decision-making.

Different synthesis methods impact interpretability.

Abstract

Understanding the interactions of agents trained with deep reinforcement learning is crucial for deploying agents in games or the real world. In the former, unreasonable actions confuse players. In the latter, that effect is even more significant, as unexpected behavior cause accidents with potentially grave and long-lasting consequences for the involved individuals. In this work, we propose using program synthesis to imitate reinforcement learning policies after seeing a trajectory of the action sequence. Programs have the advantage that they are inherently interpretable and verifiable for correctness. We adapt the state-of-the-art program synthesis system DreamCoder for learning concepts in grid-based environments, specifically, a navigation task and two miniature versions of Atari games, Space Invaders and Asterix. By inspecting the generated libraries, we can make inferences about the concepts the black-box agent has learned and better understand the agent's behavior. We achieve the same by visualizing the agent's decision-making process for the imitated sequences. We evaluate our approach with different types of program synthesizers based on a search-only method, a neural-guided search, and a language model fine-tuned on code.