Orthogonal Weight Modification Enhances Learning Scalability and Convergence Efficiency without Gradient Backpropagation

TL;DR

LOCO, a perturbation-based non-backpropagation method inspired by brain mechanisms, improves learning scalability and convergence efficiency for deep spiking neural networks with minimal parallel time complexity.

Contribution

Introduces LOCO, a low-rank orthogonal weight modification approach that enhances efficiency and scalability of non-BP learning in deep neuromorphic networks.

Findings

Enables training of networks over 10 layers deep.

Achieves better task performance than other non-BP algorithms.

Requires only O(1) parallel time complexity for weight updates.

Abstract

Recognizing the substantial computational cost of backpropagation (BP), non-BP methods have emerged as attractive alternatives for efficient learning on emerging neuromorphic systems. However, existing non-BP approaches still face critical challenges in efficiency and scalability. Inspired by neural representations and dynamic mechanisms in the brain, we propose a perturbation-based approach called LOw-rank Cluster Orthogonal (LOCO) weight modification. We find that low-rank is an inherent property of perturbation-based algorithms. Under this condition, the orthogonality constraint limits the variance of the node perturbation (NP) gradient estimates and enhances the convergence efficiency. Through extensive evaluations on multiple datasets, LOCO demonstrates the capability to locally train the deepest spiking neural networks to date (more than 10 layers), while exhibiting strong continual learning ability, improved convergence efficiency, and better task performance compared to other brain-inspired non-BP algorithms. Notably, LOCO requires only O(1) parallel time complexity for weight updates, which is significantly lower than that of BP methods. This offers a promising direction for achieving high-performance, real-time, and lifelong learning on neuromorphic systems.