A Framework for Understanding and Visualizing Strategies of RL Agents

TL;DR

This paper introduces a framework that uses temporal logic and clustering to understand and visualize strategies of reinforcement learning agents in complex environments like StarCraft II, enhancing explainability.

Contribution

The novel framework combines trace clustering with logical formula inference to produce interpretable models of agent strategies in sequential decision tasks.

Findings

Successfully separates agent behaviors into distinct clusters

Produces consistent and meaningful strategy descriptions

Enhances understanding of RL agent strategies in complex environments

Abstract

Recent years have seen significant advances in explainable AI as the need to understand deep learning models has gained importance with the increased emphasis on trust and ethics in AI. Comprehensible models for sequential decision tasks are a particular challenge as they require understanding not only individual predictions but a series of predictions that interact with environmental dynamics. We present a framework for learning comprehensible models of sequential decision tasks in which agent strategies are characterized using temporal logic formulas. Given a set of agent traces, we first cluster the traces using a novel embedding method that captures frequent action patterns. We then search for logical formulas that explain the agent strategies in the different clusters. We evaluate our framework on combat scenarios in StarCraft II (SC2), using traces from a handcrafted expert policy and a trained reinforcement learning agent. We implemented a feature extractor for SC2 environments that extracts traces as sequences of high-level features describing both the state of the environment and the agent's local behavior from agent replays. We further designed a visualization tool depicting the movement of units in the environment that helps understand how different task conditions lead to distinct agent behavior patterns in each trace cluster. Experimental results show that our framework is capable of separating agent traces into distinct groups of behaviors for which our approach to strategy inference produces consistent, meaningful, and easily understood strategy descriptions.