Learning to Perceive in Deep Model-Free Reinforcement Learning

TL;DR

This paper introduces a novel deep reinforcement learning agent with a hard attention mechanism inspired by human active perception, capable of learning to perform tasks with partial input observation, matching state-of-the-art performance in some Atari games.

Contribution

It combines a recurrent attention model with PPO to enable RL agents to focus on input regions, a novel approach not previously applied for this purpose.

Findings

Achieves comparable performance to full-input RL agents in some Atari games

Demonstrates effective attention movement similar to human behavior

Works with both discrete and continuous action spaces

Abstract

This work proposes a novel model-free Reinforcement Learning (RL) agent that is able to learn how to complete an unknown task having access to only a part of the input observation. We take inspiration from the concepts of visual attention and active perception that are characteristic of humans and tried to apply them to our agent, creating a hard attention mechanism. In this mechanism, the model decides first which region of the input image it should look at, and only after that it has access to the pixels of that region. Current RL agents do not follow this principle and we have not seen these mechanisms applied to the same purpose as this work. In our architecture, we adapt an existing model called recurrent attention model (RAM) and combine it with the proximal policy optimization (PPO) algorithm. We investigate whether a model with these characteristics is capable of achieving similar performance to state-of-the-art model-free RL agents that access the full input observation. This analysis is made in two Atari games, Pong and SpaceInvaders, which have a discrete action space, and in CarRacing, which has a continuous action space. Besides assessing its performance, we also analyze the movement of the attention of our model and compare it with what would be an example of the human behavior. Even with such visual limitation, we show that our model matches the performance of PPO+LSTM in two of the three games tested.