Revealing the Learning Process in Reinforcement Learning Agents Through Attention-Oriented Metrics

TL;DR

This paper introduces attention-oriented metrics (ATOMs) to analyze and understand the development of attention in reinforcement learning agents during training, revealing phases and behavioral correlations.

Contribution

The paper presents a novel set of attention-oriented metrics (ATOMs) for monitoring RL agents' attention development, validated through experiments on Pong variations.

Findings

ATOMs effectively distinguish attention patterns across different training phases

Attention development occurs in consistent phases across game variations

Attention patterns correlate with specific learned behaviors

Abstract

The learning process of a reinforcement learning (RL) agent remains poorly understood beyond the mathematical formulation of its learning algorithm. To address this gap, we introduce attention-oriented metrics (ATOMs) to investigate the development of an RL agent's attention during training. In a controlled experiment, we tested ATOMs on three variations of a Pong game, each designed to teach the agent distinct behaviours, complemented by a behavioural assessment. ATOMs successfully delineate the attention patterns of an agent trained on each game variation, and that these differences in attention patterns translate into differences in the agent's behaviour. Through continuous monitoring of ATOMs during training, we observed that the agent's attention developed in phases, and that these phases were consistent across game variations. Overall, we believe that ATOM could help improve our understanding of the learning processes of RL agents and better understand the relationship between attention and learning.