Frozen Backpropagation: Relaxing Weight Symmetry in Deep Spiking Neural Networks

TL;DR

This paper introduces Frozen Backpropagation (fBP), a novel training algorithm for deep spiking neural networks that relaxes the weight symmetry constraint, reducing hardware overhead and energy costs while maintaining competitive accuracy.

Contribution

fBP is the first method to relax weight symmetry in BP training for SNNs, significantly reducing weight transport costs with minimal accuracy loss.

Findings

fBP outperforms existing methods in accuracy and efficiency.

Partial weight transport schemes can reduce costs by up to 10,000x.

fBP achieves accuracy comparable to traditional BP with lower transport overhead.

Abstract

Direct training of Spiking Neural Networks (SNNs) on neuromorphic hardware can greatly reduce energy costs compared to GPU-based training. However, implementing Backpropagation (BP) on such hardware is challenging because forward and backward passes are typically performed by separate networks with distinct weights. To compute correct gradients, forward and feedback weights must remain symmetric during training, necessitating weight transport between the two networks. This symmetry requirement imposes hardware overhead and increases energy costs. To address this issue, we introduce Frozen Backpropagation (\textsc{fBP}), a BP-based training algorithm relaxing weight symmetry in settings with separate networks. fBP updates forward weights by computing gradients with periodically frozen feedback weights, reducing weight transports during training and minimizing synchronization overhead. To further improve transport efficiency, we propose three partial weight transport schemes of varying computational complexity, where only a subset of weights is transported at a time. We evaluate our methods on image recognition tasks using both temporally and rate-coded SNNs, and compare them to existing approaches addressing the weight symmetry requirement. Our results show that fBP outperforms these methods and achieves accuracy comparable to BP while significantly lowering transport costs. With partial weight transport, fBP can further lower those costs by up to 10,000x at the expense of moderate accuracy loss. This work provides insights for guiding the design of neuromorphic hardware incorporating BP-based on-chip learning.