Perception-Prediction-Reaction Agents for Deep Reinforcement Learning

TL;DR

This paper presents a novel hierarchical recurrent agent architecture with auxiliary losses that enhances reinforcement learning in partially observable environments requiring long-term memory, outperforming baseline models.

Contribution

The paper introduces the Perception-Prediction-Reaction (PPR) agent with a hierarchical structure and auxiliary losses, a new approach for improving long-term memory in reinforcement learning agents.

Findings

PPR outperforms LSTM baseline in DMLab-30 tasks.

Significant improvements in Capture the Flag environment.

Ablation studies confirm the necessity of all components.

Abstract

We introduce a new recurrent agent architecture and associated auxiliary losses which improve reinforcement learning in partially observable tasks requiring long-term memory. We employ a temporal hierarchy, using a slow-ticking recurrent core to allow information to flow more easily over long time spans, and three fast-ticking recurrent cores with connections designed to create an information asymmetry. The \emph{reaction} core incorporates new observations with input from the slow core to produce the agent's policy; the \emph{perception} core accesses only short-term observations and informs the slow core; lastly, the \emph{prediction} core accesses only long-term memory. An auxiliary loss regularizes policies drawn from all three cores against each other, enacting the prior that the policy should be expressible from either recent or long-term memory. We present the resulting \emph{Perception-Prediction-Reaction} (PPR) agent and demonstrate its improved performance over a strong LSTM-agent baseline in DMLab-30, particularly in tasks requiring long-term memory. We further show significant improvements in Capture the Flag, an environment requiring agents to acquire a complicated mixture of skills over long time scales. In a series of ablation experiments, we probe the importance of each component of the PPR agent, establishing that the entire, novel combination is necessary for this intriguing result.