Efficient Deep Reinforcement Learning with Predictive Processing Proximal Policy Optimization

TL;DR

This paper introduces P4O, a reinforcement learning agent that integrates predictive processing inspired by neuroscience, leading to significant improvements in sample efficiency and performance on Atari games, surpassing human levels in some cases.

Contribution

The paper presents P4O, a novel RL agent combining predictive processing with PPO, demonstrating enhanced efficiency and performance without hyperparameter tuning.

Findings

P4O outperforms baseline recurrent PPO on Atari games.

P4O surpasses state-of-the-art agents within the same training time.

P4O exceeds human performance on multiple challenging Atari games.

Abstract

Advances in reinforcement learning (RL) often rely on massive compute resources and remain notoriously sample inefficient. In contrast, the human brain is able to efficiently learn effective control strategies using limited resources. This raises the question whether insights from neuroscience can be used to improve current RL methods. Predictive processing is a popular theoretical framework which maintains that the human brain is actively seeking to minimize surprise. We show that recurrent neural networks which predict their own sensory states can be leveraged to minimise surprise, yielding substantial gains in cumulative reward. Specifically, we present the Predictive Processing Proximal Policy Optimization (P4O) agent; an actor-critic reinforcement learning agent that applies predictive processing to a recurrent variant of the PPO algorithm by integrating a world model in its hidden state. Even without hyperparameter tuning, P4O significantly outperforms a baseline recurrent variant of the PPO algorithm on multiple Atari games using a single GPU. It also outperforms other state-of-the-art agents given the same wall-clock time and exceeds human gamer performance on multiple games including Seaquest, which is a particularly challenging environment in the Atari domain. Altogether, our work underscores how insights from the field of neuroscience may support the development of more capable and efficient artificial agents.