Adaptive Surrogate Gradients for Sequential Reinforcement Learning in Spiking Neural Networks

TL;DR

This paper introduces adaptive surrogate gradients and a guiding policy to improve training of spiking neural networks for reinforcement learning, achieving significant performance gains in real-world robotic control.

Contribution

It provides a systematic analysis of surrogate gradient slopes and proposes an adaptive slope schedule combined with a guiding policy for effective RL training in SNNs.

Findings

Shallower slopes increase gradient magnitude in deep layers but reduce alignment.

Shallow or scheduled slopes improve RL performance by 2.1x.

The proposed method outperforms prior techniques in drone control, achieving an average return of 400 points.

Abstract

Neuromorphic computing systems are set to revolutionize energy-constrained robotics by achieving orders-of-magnitude efficiency gains, while enabling native temporal processing. Spiking Neural Networks (SNNs) represent a promising algorithmic approach for these systems, yet their application to complex control tasks faces two critical challenges: (1) the non-differentiable nature of spiking neurons necessitates surrogate gradients with unclear optimization properties, and (2) the stateful dynamics of SNNs require training on sequences, which in reinforcement learning (RL) is hindered by limited sequence lengths during early training, preventing the network from bridging its warm-up period. We address these challenges by systematically analyzing surrogate gradient slope settings, showing that shallower slopes increase gradient magnitude in deeper layers but reduce alignment with true gradients. In supervised learning, we find no clear preference for fixed or scheduled slopes. The effect is much more pronounced in RL settings, where shallower slopes or scheduled slopes lead to a 2.1x improvement in both training and final deployed performance. Next, we propose a novel training approach that leverages a privileged guiding policy to bootstrap the learning process, while still exploiting online environment interactions with the spiking policy. Combining our method with an adaptive slope schedule for a real-world drone position control task, we achieve an average return of 400 points, substantially outperforming prior techniques, including Behavioral Cloning and TD3BC, which achieve at most --200 points under the same conditions. This work advances both the theoretical understanding of surrogate gradient learning in SNNs and practical training methodologies for neuromorphic controllers demonstrated in real-world robotic systems.