CADE: Cosine Annealing Differential Evolution for Spiking Neural Network

TL;DR

This paper introduces CADE, a novel optimization method that combines cosine annealing with differential evolution to improve the training of spiking neural networks, achieving faster convergence and higher accuracy.

Contribution

It proposes a cosine annealing strategy for differential evolution parameters and an initialization method using transfer learning to enhance SNN training.

Findings

CADE accelerates convergence and improves accuracy over existing methods.

Transfer learning-based initialization boosts population diversity and model performance.

CADE increases the highest accuracy of SEW SNN models by 0.52 percentage points.

Abstract

Spiking neural networks (SNNs) have gained prominence for their potential in neuromorphic computing and energy-efficient artificial intelligence, yet optimizing them remains a formidable challenge for gradient-based methods due to their discrete, spike-based computation. This paper attempts to tackle the challenges by introducing Cosine Annealing Differential Evolution (CADE), designed to modulate the mutation factor (F) and crossover rate (CR) of differential evolution (DE) for the SNN model, i.e., Spiking Element Wise (SEW) ResNet. Extensive empirical evaluations were conducted to analyze CADE. CADE showed a balance in exploring and exploiting the search space, resulting in accelerated convergence and improved accuracy compared to existing gradient-based and DE-based methods. Moreover, an initialization method based on a transfer learning setting was developed, pretraining on a source dataset (i.e., CIFAR-10) and fine-tuning the target dataset (i.e., CIFAR-100), to improve population diversity. It was found to further enhance CADE for SNN. Remarkably, CADE elevates the performance of the highest accuracy SEW model by an additional 0.52 percentage points, underscoring its effectiveness in fine-tuning and enhancing SNNs. These findings emphasize the pivotal role of a scheduler for F and CR adjustment, especially for DE-based SNN. Source Code on Github: https://github.com/Tank-Jiang/CADE4SNN.