Efficient Nonlinear Function Approximation in Analog Resistive Crossbars for Recurrent Neural Networks

TL;DR

This paper presents a novel analog in-memory computing method using memristive arrays to efficiently implement nonlinear activation functions in RNNs, significantly improving energy, area, and throughput performance.

Contribution

It introduces a new ramp-based analog approach with memristors for nonlinear functions, enhancing RNN implementation efficiency in analog IMC systems.

Findings

Successful experimental implementation of nonlinear functions with memristors.

Improved RNN performance on keyword spotting and language modeling tasks.

Significant gains in area, energy, and throughput efficiency.

Abstract

Analog In-memory Computing (IMC) has demonstrated energy-efficient and low latency implementation of convolution and fully-connected layers in deep neural networks (DNN) by using physics for computing in parallel resistive memory arrays. However, recurrent neural networks (RNN) that are widely used for speech-recognition and natural language processing have tasted limited success with this approach. This can be attributed to the significant time and energy penalties incurred in implementing nonlinear activation functions that are abundant in such models. In this work, we experimentally demonstrate the implementation of a non-linear activation function integrated with a ramp analog-to-digital conversion (ADC) at the periphery of the memory to improve in-memory implementation of RNNs. Our approach uses an extra column of memristors to produce an appropriately pre-distorted ramp voltage such that the comparator output directly approximates the desired nonlinear function. We experimentally demonstrate programming different nonlinear functions using a memristive array and simulate its incorporation in RNNs to solve keyword spotting and language modelling tasks. Compared to other approaches, we demonstrate manifold increase in area-efficiency, energy-efficiency and throughput due to the in-memory, programmable ramp generator that removes digital processing overhead.