Backprop-Free Reinforcement Learning with Active Neural Generative Coding

TL;DR

This paper introduces active neural generative coding, a backpropagation-free method for reinforcement learning that enables agents to learn and act effectively in dynamic environments with sparse rewards.

Contribution

It presents a novel backprop-free framework for neural inference and learning, inspired by cognitive planning as inference, applicable to reinforcement learning tasks.

Findings

Performs competitively with deep Q-learning on control problems

Operates effectively with sparse rewards

Supports goal-directed behavior without backpropagation

Abstract

In humans, perceptual awareness facilitates the fast recognition and extraction of information from sensory input. This awareness largely depends on how the human agent interacts with the environment. In this work, we propose active neural generative coding, a computational framework for learning action-driven generative models without backpropagation of errors (backprop) in dynamic environments. Specifically, we develop an intelligent agent that operates even with sparse rewards, drawing inspiration from the cognitive theory of planning as inference. We demonstrate on several simple control problems that our framework performs competitively with deep Q-learning. The robust performance of our agent offers promising evidence that a backprop-free approach for neural inference and learning can drive goal-directed behavior.