Toward Large-scale Spiking Neural Networks: A Comprehensive Survey and Future Directions

TL;DR

This survey reviews current methods and architectures for deep spiking neural networks, emphasizing energy efficiency and future directions for large-scale models, including emerging Spiking Transformers.

Contribution

It provides a comprehensive overview of learning techniques, architectures, and comparisons of deep SNNs, highlighting future research directions for large-scale energy-efficient models.

Findings

Deep SNNs can be developed via ANN-to-SNN conversion or surrogate gradient training.

Spiking Transformers are emerging as a promising architecture for deep SNNs.

Comparison shows competitive performance of SNNs with traditional neural networks.

Abstract

Deep learning has revolutionized artificial intelligence (AI), achieving remarkable progress in fields such as computer vision, speech recognition, and natural language processing. Moreover, the recent success of large language models (LLMs) has fueled a surge in research on large-scale neural networks. However, the escalating demand for computing resources and energy consumption has prompted the search for energy-efficient alternatives. Inspired by the human brain, spiking neural networks (SNNs) promise energy-efficient computation with event-driven spikes. To provide future directions toward building energy-efficient large SNN models, we present a survey of existing methods for developing deep spiking neural networks, with a focus on emerging Spiking Transformers. Our main contributions are as follows: (1) an overview of learning methods for deep spiking neural networks, categorized by ANN-to-SNN conversion and direct training with surrogate gradients; (2) an overview of network architectures for deep spiking neural networks, categorized by deep convolutional neural networks (DCNNs) and Transformer architecture; and (3) a comprehensive comparison of state-of-the-art deep SNNs with a focus on emerging Spiking Transformers. We then further discuss and outline future directions toward large-scale SNNs.