Trapezoidal Gradient Descent for Effective Reinforcement Learning in Spiking Networks

TL;DR

This paper introduces a trapezoidal gradient approximation method for reinforcement learning in spiking neural networks, improving training sensitivity, convergence speed, and stability while reducing energy consumption.

Contribution

It proposes a novel trapezoidal approximation gradient technique that enhances SNN training effectiveness over existing rectangular function-based methods.

Findings

Faster convergence compared to original algorithms

Improved training stability and adaptability

Enhanced response sensitivity under various signals

Abstract

With the rapid development of artificial intelligence technology, the field of reinforcement learning has continuously achieved breakthroughs in both theory and practice. However, traditional reinforcement learning algorithms often entail high energy consumption during interactions with the environment. Spiking Neural Network (SNN), with their low energy consumption characteristics and performance comparable to deep neural networks, have garnered widespread attention. To reduce the energy consumption of practical applications of reinforcement learning, researchers have successively proposed the Pop-SAN and MDC-SAN algorithms. Nonetheless, these algorithms use rectangular functions to approximate the spike network during the training process, resulting in low sensitivity, thus indicating room for improvement in the training effectiveness of SNN. Based on this, we propose a trapezoidal approximation gradient method to replace the spike network, which not only preserves the original stable learning state but also enhances the model's adaptability and response sensitivity under various signal dynamics. Simulation results show that the improved algorithm, using the trapezoidal approximation gradient to replace the spike network, achieves better convergence speed and performance compared to the original algorithm and demonstrates good training stability.