Bellman Memory Units: A neuromorphic framework for synaptic reinforcement learning with an evolving network topology

TL;DR

This paper presents a neuromorphic framework using Bellman Memory Units for synaptic reinforcement learning, enabling evolving network topologies and on-chip adaptation on neuromorphic hardware like Intel's Loihi.

Contribution

It introduces a synaptic Q-learning algorithm with evolving topology and implements neuromorphic Bellman Memory Units on Loihi for scalable, resource-efficient reinforcement learning.

Findings

Topology evolution improves learning efficiency

On-chip learning enables adaptation to new scenarios

Resource optimization reduces hardware complexity

Abstract

Application of neuromorphic edge devices for control is limited by the constraints on gradient-free online learning and scalability of the hardware across control problems. This paper introduces a synaptic Q-learning algorithm for the control of the classical Cartpole, where the Bellman equations are incorporated at the synaptic level. This formulation enables the iterative evolution of the network topology, represented as a directed graph, throughout the training process. This is followed by a similar approach called neuromorphic Bellman Memory Units (BMU(s)), which are implemented with the Neural Engineering Framework on Intel's Loihi neuromorphic chip. Topology evolution, in conjunction with mixed-signal computation, leverages the optimization of the number of neurons and synapses that could be used to design spike-based reinforcement learning accelerators. The proposed architecture can potentially reduce resource utilization on board, aiding the manufacturing of compact application-specific neuromorphic ICs. Moreover, the on-chip learning introduced in this work and implemented on a neuromorphic chip can enable adaptation to unseen control scenarios.