Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning

TL;DR

This paper introduces ILC-SAN, a fully spiking actor network with intra-layer connections that enables energy-efficient reinforcement learning without floating-point operations, inspired by insect neural structures.

Contribution

It proposes a novel fully spiking actor network architecture with intra-layer connections, eliminating the need for floating-point operations and enabling deployment on neuromorphic hardware.

Findings

Achieves comparable performance to traditional methods in control tasks.

Enables direct deployment on neuromorphic hardware due to fully spiking design.

Enhances representation capacity through intra-layer connections.

Abstract

With the help of special neuromorphic hardware, spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (DRL). In this paper, we focus on the task where the agent needs to learn multi-dimensional deterministic policies to control, which is very common in real scenarios. Recently, the surrogate gradient method has been utilized for training multi-layer SNNs, which allows SNNs to achieve comparable performance with the corresponding deep networks in this task. Most existing spike-based RL methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully-connected (FC) layer. However, the decimal characteristic of the firing rate brings the floating-point matrix operations to the FC layer, making the whole SNN unable to deploy on the neuromorphic hardware directly. To develop a fully spiking actor network without any floating-point matrix operations, we draw inspiration from the non-spiking interneurons found in insects and employ the membrane voltage of the non-spiking neurons to represent the action. Before the non-spiking neurons, multiple population neurons are introduced to decode different dimensions of actions. Since each population is used to decode a dimension of action, we argue that the neurons in each population should be connected in time domain and space domain. Hence, the intra-layer connections are used in output populations to enhance the representation capacity. Finally, we propose a fully spiking actor network with intra-layer connections (ILC-SAN).