Signal-Adaptive Trust Regions for Gradient-Free Optimization of Recurrent Spiking Neural Networks

TL;DR

This paper introduces Signal-Adaptive Trust Regions (SATR), a novel gradient-free optimization method for training recurrent spiking neural networks, which adaptively constrains policy updates based on signal strength, improving stability and efficiency in reinforcement learning tasks.

Contribution

The paper proposes SATR, a new trust-region method that adapts to signal energy for stable gradient-free training of RSNNs, and introduces a bitset implementation for scalable, fast policy search.

Findings

SATR improves training stability with limited populations.

SATR achieves competitive performance on continuous-control benchmarks.

Bitset implementation significantly reduces training time.

Abstract

Recurrent spiking neural networks (RSNNs) are a promising substrate for energy-efficient control policies, but training them for high-dimensional, long-horizon reinforcement learning remains challenging. Population-based, gradient-free optimization circumvents backpropagation through non-differentiable spike dynamics by estimating gradients. However, with finite populations, high variance of these estimates can induce harmful and overly aggressive update steps. Inspired by trust-region methods in reinforcement learning that constrain policy updates in distribution space, we propose \textbf{Signal-Adaptive Trust Regions (SATR)}, a distributional update rule that constrains relative change by bounding KL divergence normalized by an estimated signal energy. SATR automatically expands the trust region under strong signals and contracts it when updates are noise-dominated. We instantiate SATR for Bernoulli connectivity distributions, which have shown strong empirical performance for RSNN optimization. Across a suite of high-dimensional continuous-control benchmarks, SATR improves stability under limited populations and reaches competitive returns against strong baselines including PPO-LSTM. In addition, to make SATR practical at scale, we introduce a bitset implementation for binary spiking and binary weights, substantially reducing wall-clock training time and enabling fast RSNN policy search.