TripleTree: A Versatile Interpretable Representation of Black Box Agents and their Environments

TL;DR

This paper introduces TripleTree, a novel decision tree-based method for creating interpretable, convex-region representations of black box agents and their environments, aiding understanding through prediction, visualization, and explanations.

Contribution

It presents a new variant of CART decision trees that discretizes state spaces into convex regions, enabling interpretability of complex reinforcement learning agents.

Findings

Effective in predicting agent behavior

Enhances visualization of agent decision processes

Provides rule-based explanations for black box agents

Abstract

In explainable artificial intelligence, there is increasing interest in understanding the behaviour of autonomous agents to build trust and validate performance. Modern agent architectures, such as those trained by deep reinforcement learning, are currently so lacking in interpretable structure as to effectively be black boxes, but insights may still be gained from an external, behaviourist perspective. Inspired by conceptual spaces theory, we suggest that a versatile first step towards general understanding is to discretise the state space into convex regions, jointly capturing similarities over the agent's action, value function and temporal dynamics within a dataset of observations. We create such a representation using a novel variant of the CART decision tree algorithm, and demonstrate how it facilitates practical understanding of black box agents through prediction, visualisation and rule-based explanation.