Temporal-Spatial Causal Interpretations for Vision-Based Reinforcement Learning

TL;DR

This paper introduces TSCI, a novel method for interpreting vision-based reinforcement learning agents by uncovering temporal-spatial causal features, thereby enhancing understanding of their long-term decision-making processes.

Contribution

The paper proposes a temporal-spatial causal interpretation model that captures long-term causal relations in vision-based RL, addressing limitations of existing methods.

Findings

Produces high-resolution, task-relevant attention masks

Effectively uncovers temporal causal features in RL agents

Applicable to recurrent RL agents for causal discovery

Abstract

Deep reinforcement learning (RL) agents are becoming increasingly proficient in a range of complex control tasks. However, the agent's behavior is usually difficult to interpret due to the introduction of black-box function, making it difficult to acquire the trust of users. Although there have been some interesting interpretation methods for vision-based RL, most of them cannot uncover temporal causal information, raising questions about their reliability. To address this problem, we present a temporal-spatial causal interpretation (TSCI) model to understand the agent's long-term behavior, which is essential for sequential decision-making. TSCI model builds on the formulation of temporal causality, which reflects the temporal causal relations between sequential observations and decisions of RL agent. Then a separate causal discovery network is employed to identify temporal-spatial causal features, which are constrained to satisfy the temporal causality. TSCI model is applicable to recurrent agents and can be used to discover causal features with high efficiency once trained. The empirical results show that TSCI model can produce high-resolution and sharp attention masks to highlight task-relevant temporal-spatial information that constitutes most evidence about how vision-based RL agents make sequential decisions. In addition, we further demonstrate that our method is able to provide valuable causal interpretations for vision-based RL agents from the temporal perspective.