Global and Local Analysis of Interestingness for Competency-Aware Deep Reinforcement Learning

TL;DR

This paper introduces a comprehensive framework for analyzing and interpreting the competence of reinforcement learning agents through interestingness measures, enabling better understanding of their capabilities and limitations.

Contribution

It extends explainable RL with new interestingness-based analysis tools, including behavior clustering and task element attribution using SHAP values, applicable across various RL algorithms.

Findings

Provides measures of RL agent competence based on interestingness analysis.

Introduces clustering of behavior traces to identify competency patterns.

Uses SHAP values for global and local analysis of task elements influencing behavior.

Abstract

In recent years, advances in deep learning have resulted in a plethora of successes in the use of reinforcement learning (RL) to solve complex sequential decision tasks with high-dimensional inputs. However, existing systems lack the necessary mechanisms to provide humans with a holistic view of their competence, presenting an impediment to their adoption, particularly in critical applications where the decisions an agent makes can have significant consequences. Yet, existing RL-based systems are essentially competency-unaware in that they lack the necessary interpretation mechanisms to allow human operators to have an insightful, holistic view of their competency. In this paper, we extend a recently-proposed framework for explainable RL that is based on analyses of "interestingness." Our new framework provides various measures of RL agent competence stemming from interestingness analysis and is applicable to a wide range of RL algorithms. We also propose novel mechanisms for assessing RL agents' competencies that: 1) identify agent behavior patterns and competency-controlling conditions by clustering agent behavior traces solely using interestingness data; and 2) identify the task elements mostly responsible for an agent's behavior, as measured through interestingness, by performing global and local analyses using SHAP values. Overall, our tools provide insights about RL agent competence, both their capabilities and limitations, enabling users to make more informed decisions about interventions, additional training, and other interactions in collaborative human-machine settings.