Modelling Agent Policies with Interpretable Imitation Learning

TL;DR

This paper introduces an imitation learning approach that creates interpretable decision tree models of agent policies, enabling understanding of black box agents' internal states in safety-critical domains.

Contribution

It presents a novel method for reverse-engineering black box policies into decision trees and explicitly models latent state representations from Markov states.

Findings

Initial results in a multi-agent traffic environment are promising.

The approach produces simplified, interpretable models of complex policies.

Explicit latent state modeling enhances understanding of agent behavior.

Abstract

As we deploy autonomous agents in safety-critical domains, it becomes important to develop an understanding of their internal mechanisms and representations. We outline an approach to imitation learning for reverse-engineering black box agent policies in MDP environments, yielding simplified, interpretable models in the form of decision trees. As part of this process, we explicitly model and learn agents' latent state representations by selecting from a large space of candidate features constructed from the Markov state. We present initial promising results from an implementation in a multi-agent traffic environment.