Efficient Training of the Memristive Deep Belief Net Immune to Non-Idealities of the Synaptic Devices

TL;DR

This paper introduces an efficient memristive deep belief network training method that is robust to device non-idealities, reduces write operations, and achieves high accuracy on MNIST.

Contribution

It proposes a mixed-signal hardware approach with stochastic binarization and contrastive divergence for robust, low-complexity memristive DBN training.

Findings

Achieves 95-97% accuracy on MNIST dataset.

Reduces memristive device write operations by two orders of magnitude.

Demonstrates robustness to non-idealities of memristive devices.

Abstract

The tunability of conductance states of various emerging non-volatile memristive devices emulates the plasticity of biological synapses, making it promising in the hardware realization of large-scale neuromorphic systems. The inference of the neural network can be greatly accelerated by the vector-matrix multiplication (VMM) performed within a crossbar array of memristive devices in one step. Nevertheless, the implementation of the VMM needs complex peripheral circuits and the complexity further increases since non-idealities of memristive devices prevent precise conductance tuning (especially for the online training) and largely degrade the performance of the deep neural networks (DNNs). Here, we present an efficient online training method of the memristive deep belief net (DBN). The proposed memristive DBN uses stochastically binarized activations, reducing the complexity of peripheral circuits, and uses the contrastive divergence (CD) based gradient descent learning algorithm. The analog VMM and digital CD are performed separately in a mixed-signal hardware arrangement, making the memristive DBN high immune to non-idealities of synaptic devices. The number of write operations on memristive devices is reduced by two orders of magnitude. The recognition accuracy of 95%~97% can be achieved for the MNIST dataset using pulsed synaptic behaviors of various memristive synaptic devices.