Explaining Imitation Learning through Frames

TL;DR

This paper introduces R2RISE, a model-agnostic framework that explains imitation learning policies by identifying important frames in demonstrations, enhancing interpretability of black-box IL models.

Contribution

It presents a novel explainability method for IL models that uses randomized masking and importance maps to interpret policy decisions.

Findings

R2RISE effectively identifies important frames in demonstrations.

The importance maps correlate with policy performance.

The framework reveals differences in frame importance across models.

Abstract

As one of the prevalent methods to achieve automation systems, Imitation Learning (IL) presents a promising performance in a wide range of domains. However, despite the considerable improvement in policy performance, the corresponding research on the explainability of IL models is still limited. Inspired by the recent approaches in explainable artificial intelligence methods, we proposed a model-agnostic explaining framework for IL models called R2RISE. R2RISE aims to explain the overall policy performance with respect to the frames in demonstrations. It iteratively retrains the black-box IL model from the randomized masked demonstrations and uses the conventional evaluation outcome environment returns as the coefficient to build an importance map. We also conducted experiments to investigate three major questions concerning frames' importance equality, the effectiveness of the importance map, and connections between importance maps from different IL models. The result shows that R2RISE successfully distinguishes important frames from the demonstrations.