Understanding the Convergence in Balanced Resonate-and-Fire Neurons

TL;DR

This paper investigates why balanced resonate-and-fire neurons exhibit faster and smoother convergence in training SNNs, highlighting their nearly convex error landscape and the role of divergence boundary-aware optimization.

Contribution

It provides new insights into the convergence advantages of BRF neurons, emphasizing their smooth error landscape and stability mechanisms compared to ALIF neurons.

Findings

BRF neurons have a nearly convex error landscape.

Convergence benefits are linked to divergence boundary-aware optimization.

Membrane dynamics allow gradient transfer without loss.

Abstract

Resonate-and-Fire (RF) neurons are an interesting complementary model for integrator neurons in spiking neural networks (SNNs). Due to their resonating membrane dynamics they can extract frequency patterns within the time domain. While established RF variants suffer from intrinsic shortcomings, the recently proposed balanced resonate-and-fire (BRF) neuron marked a significant methodological advance in terms of task performance, spiking and parameter efficiency, as well as, general stability and robustness, demonstrated for recurrent SNNs in various sequence learning tasks. One of the most intriguing result, however, was an immense improvement in training convergence speed and smoothness, overcoming the typical convergence dilemma in backprop-based SNN training. This paper aims at providing further intuitions about how and why these convergence advantages emerge. We show that BRF neurons, in contrast to well-established ALIF neurons, span a very clean and smooth - almost convex - error landscape. Furthermore, empirical results reveal that the convergence benefits are predominantly coupled with a divergence boundary-aware optimization, a major component of the BRF formulation that addresses the numerical stability of the time-discrete resonator approximation. These results are supported by a formal investigation of the membrane dynamics indicating that the gradient is transferred back through time without loss of magnitude.