Neuromorphic dreaming: A pathway to efficient learning in artificial agents

TL;DR

This paper introduces a neuromorphic hardware system implementing model-based reinforcement learning with spiking neural networks, achieving energy-efficient learning and reduced real experience requirements in training an Atari game agent.

Contribution

It presents a novel neuromorphic hardware implementation of MBRL with dreaming phases, combining online and offline learning for improved efficiency.

Findings

Successfully trained on Atari Pong using neuromorphic hardware

Dreaming reduces the number of real experiences needed for learning

Hardware implementation demonstrates energy-efficient learning capabilities

Abstract

Achieving energy efficiency in learning is a key challenge for artificial intelligence (AI) computing platforms. Biological systems demonstrate remarkable abilities to learn complex skills quickly and efficiently. Inspired by this, we present a hardware implementation of model-based reinforcement learning (MBRL) using spiking neural networks (SNNs) on mixed-signal analog/digital neuromorphic hardware. This approach leverages the energy efficiency of mixed-signal neuromorphic chips while achieving high sample efficiency through an alternation of online learning, referred to as the "awake" phase, and offline learning, known as the "dreaming" phase. The model proposed includes two symbiotic networks: an agent network that learns by combining real and simulated experiences, and a learned world model network that generates the simulated experiences. We validate the model by training the hardware implementation to play the Atari game Pong. We start from a baseline consisting of an agent network learning without a world model and dreaming, which successfully learns to play the game. By incorporating dreaming, the number of required real game experiences are reduced significantly compared to the baseline. The networks are implemented using a mixed-signal neuromorphic processor, with the readout layers trained using a computer in-the-loop, while the other layers remain fixed. These results pave the way toward energy-efficient neuromorphic learning systems capable of rapid learning in real world applications and use-cases.