Action Q-Transformer: Visual Explanation in Deep Reinforcement Learning with Encoder-Decoder Model using Action Query

TL;DR

The paper introduces the Action Q-Transformer, a novel encoder-decoder transformer model for deep reinforcement learning that enhances interpretability through attention visualization while maintaining or improving performance.

Contribution

It proposes a transformer-based DRL model with attention visualization using action queries, improving interpretability without sacrificing performance.

Findings

Attention visualization reveals decision-making processes.

Achieves higher performance than baseline in some Atari games.

Provides detailed analysis of agent behavior in complex tasks.

Abstract

The excellent performance of Transformer in supervised learning has led to growing interest in its potential application to deep reinforcement learning (DRL) to achieve high performance on a wide variety of problems. However, the decision making of a DRL agent is a black box, which greatly hinders the application of the agent to real-world problems. To address this problem, we propose the Action Q-Transformer (AQT), which introduces a transformer encoder-decoder structure to Q-learning based DRL methods. In AQT, the encoder calculates the state value function and the decoder calculates the advantage function to promote the acquisition of different attentions indicating the agent's decision-making. The decoder in AQT utilizes action queries, which represent the information of each action, as queries. This enables us to obtain the attentions for the state value and for each action. By acquiring and visualizing these attentions that detail the agent's decision-making, we achieve a DRL model with high interpretability. In this paper, we show that visualization of attention in Atari 2600 games enables detailed analysis of agents' decision-making in various game tasks. Further, experimental results demonstrate that our method can achieve higher performance than the baseline in some games.