A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations

TL;DR

This paper introduces a neuromorphic SNN architecture for reinforcement learning with real-valued observations, demonstrating hardware-efficient online learning and superior performance on classic control tasks.

Contribution

It presents a novel bio-inspired SNN model with TD-error modulation and eligibility traces, advancing hardware-efficient RL implementations.

Findings

Outperforms tabular actor-critic in RL benchmarks

Successfully learns stable policies in classic control environments

Offers a hardware-friendly online learning approach

Abstract

Reinforcement Learning (RL) provides a powerful framework for decision-making in complex environments. However, implementing RL in hardware-efficient and bio-inspired ways remains a challenge. This paper presents a novel Spiking Neural Network (SNN) architecture for solving RL problems with real-valued observations. The proposed model incorporates multi-layered event-based clustering, with the addition of Temporal Difference (TD)-error modulation and eligibility traces, building upon prior work. An ablation study confirms the significant impact of these components on the proposed model's performance. A tabular actor-critic algorithm with eligibility traces and a state-of-the-art Proximal Policy Optimization (PPO) algorithm are used as benchmarks. Our network consistently outperforms the tabular approach and successfully discovers stable control policies on classic RL environments: mountain car, cart-pole, and acrobot. The proposed model offers an appealing trade-off in terms of computational and hardware implementation requirements. The model does not require an external memory buffer nor a global error gradient computation, and synaptic updates occur online, driven by local learning rules and a broadcasted TD-error signal. Thus, this work contributes to the development of more hardware-efficient RL solutions.