Dive into the Power of Neuronal Heterogeneity

TL;DR

This paper explores the benefits of neuronal heterogeneity in neural networks, demonstrating that diverse neuronal properties improve performance and biological plausibility, especially in complex tasks.

Contribution

It introduces a robust optimization method for heterogeneous neurons in SNNs using Evolutionary Strategy, highlighting the importance of membrane time constants.

Findings

Neuronal heterogeneity enhances performance in long sequence tasks

Membrane time constants distribution aligns with biological data

Heterogeneous networks outperform homogeneous ones in various tasks

Abstract

The biological neural network is a vast and diverse structure with high neural heterogeneity. Conventional Artificial Neural Networks (ANNs) primarily focus on modifying the weights of connections through training while modeling neurons as highly homogenized entities and lacking exploration of neural heterogeneity. Only a few studies have addressed neural heterogeneity by optimizing neuronal properties and connection weights to ensure network performance. However, this strategy impact the specific contribution of neuronal heterogeneity. In this paper, we first demonstrate the challenges faced by backpropagation-based methods in optimizing Spiking Neural Networks (SNNs) and achieve more robust optimization of heterogeneous neurons in random networks using an Evolutionary Strategy (ES). Experiments on tasks such as working memory, continuous control, and image recognition show that neuronal heterogeneity can improve performance, particularly in long sequence tasks. Moreover, we find that membrane time constants play a crucial role in neural heterogeneity, and their distribution is similar to that observed in biological experiments. Therefore, we believe that the neglected neuronal heterogeneity plays an essential role, providing new approaches for exploring neural heterogeneity in biology and new ways for designing more biologically plausible neural networks.