Speeding up reinforcement learning by combining attention and agency features

TL;DR

This paper introduces combined attention and agency detection mechanisms in reinforcement learning to improve learning speed and input-size independence, especially in pixel-based environments like Atari games.

Contribution

It proposes novel input transformations that integrate attention and agency detection, and benchmarks architectures with global and local agency-centered states.

Findings

Global-local state networks learn faster than global alone.

Summarized states show potential for input-size independent learning.

Combined mechanisms improve training efficiency.

Abstract

When playing video-games we immediately detect which entity we control and we center the attention towards it to focus the learning and reduce its dimensionality. Reinforcement Learning (RL) has been able to deal with big state spaces, including states derived from pixel images in Atari games, but the learning is slow, depends on the brute force mapping from the global state to the action values (Q-function), thus its performance is severely affected by the dimensionality of the state and cannot be transferred to other games or other parts of the same game. We propose different transformations of the input state that combine attention and agency detection mechanisms which both have been addressed separately in RL but not together to our knowledge. We propose and benchmark different architectures including both global and local agency centered versions of the state and also including summaries of the surroundings. Results suggest that even a redundant global-local state network can learn faster than the global alone. Summarized versions of the state look promising to achieve input-size independence learning.