Mapping high-performance RNNs to in-memory neuromorphic chips

TL;DR

This paper introduces an adaptive spiking neuron model that enhances training and inference efficiency of RNNs on neuromorphic chips, achieving significant energy savings over traditional microprocessors.

Contribution

A novel low-pass filter-based spiking neuron model that enables faster training and improved performance of RNNs on neuromorphic hardware.

Findings

Dramatic improvement in inference performance on complex tasks

At least 500x higher energy-efficiency compared to Cortex-M4

Effective training of RNNs without spike simulation

Abstract

The increasing need for compact and low-power computing solutions for machine learning applications has triggered significant interest in energy-efficient neuromorphic systems. However, most of these architectures rely on spiking neural networks, which typically perform poorly compared to their non-spiking counterparts in terms of accuracy. In this paper, we propose a new adaptive spiking neuron model that can be abstracted as a low-pass filter. This abstraction enables faster and better training of spiking networks using back-propagation, without simulating spikes. We show that this model dramatically improves the inference performance of a recurrent neural network and validate it with three complex spatio-temporal learning tasks: the temporal addition task, the temporal copying task, and a spoken-phrase recognition task. We estimate at least 500x higher energy-efficiency using our models on compatible neuromorphic chips in comparison to Cortex-M4, a popular embedded microprocessor.