Differentiable architecture search with multi-dimensional attention for spiking neural networks

TL;DR

This paper introduces MA-DARTS, a neural architecture search method with multi-dimensional attention, optimizing SNN structures for better accuracy and efficiency, achieving state-of-the-art results on CIFAR datasets.

Contribution

The paper proposes a novel differentiable NAS framework with multi-dimensional attention specifically designed for SNNs, improving their performance and efficiency.

Findings

Achieved 94.40% accuracy on CIFAR10

Achieved 76.52% accuracy on CIFAR100

Stabilized spike count around 110K in validation

Abstract

Spiking Neural Networks (SNNs) have gained enormous popularity in the field of artificial intelligence due to their low power consumption. However, the majority of SNN methods directly inherit the structure of Artificial Neural Networks (ANN), usually leading to sub-optimal model performance in SNNs. To alleviate this problem, we integrate Neural Architecture Search (NAS) method and propose Multi-Attention Differentiable Architecture Search (MA-DARTS) to directly automate the search for the optimal network structure of SNNs. Initially, we defined a differentiable two-level search space and conducted experiments within micro architecture under a fixed layer. Then, we incorporated a multi-dimensional attention mechanism and implemented the MA-DARTS algorithm in this search space. Comprehensive experiments demonstrate our model achieves state-of-the-art performance on classification compared to other methods under the same parameters with 94.40% accuracy on CIFAR10 dataset and 76.52% accuracy on CIFAR100 dataset. Additionally, we monitored and assessed the number of spikes (NoS) in each cell during the whole experiment. Notably, the number of spikes of the whole model stabilized at approximately 110K in validation and 100k in training on datasets.