Training Deep Neural Networks with Constrained Learning Parameters

TL;DR

This paper introduces CoNNTrA, a new algorithm for training deep neural networks with finite discrete parameters, enabling models that are memory-efficient and suitable for edge computing, without sacrificing accuracy.

Contribution

The paper proposes CoNNTrA, a coordinate gradient descent-based training method for discrete-parameter neural networks, with theoretical analysis and empirical validation on multiple datasets.

Findings

Models trained with CoNNTrA use 32x less memory than traditional methods.

CoNNTrA-trained models achieve comparable error rates to backpropagation.

CoNNTrA demonstrates feasibility for low-power, memory-constrained edge devices.

Abstract

Today's deep learning models are primarily trained on CPUs and GPUs. Although these models tend to have low error, they consume high power and utilize large amount of memory owing to double precision floating point learning parameters. Beyond the Moore's law, a significant portion of deep learning tasks would run on edge computing systems, which will form an indispensable part of the entire computation fabric. Subsequently, training deep learning models for such systems will have to be tailored and adopted to generate models that have the following desirable characteristics: low error, low memory, and low power. We believe that deep neural networks (DNNs), where learning parameters are constrained to have a set of finite discrete values, running on neuromorphic computing systems would be instrumental for intelligent edge computing systems having these desirable characteristics. To this extent, we propose the Combinatorial Neural Network Training Algorithm (CoNNTrA), that leverages a coordinate gradient descent-based approach for training deep learning models with finite discrete learning parameters. Next, we elaborate on the theoretical underpinnings and evaluate the computational complexity of CoNNTrA. As a proof of concept, we use CoNNTrA to train deep learning models with ternary learning parameters on the MNIST, Iris and ImageNet data sets and compare their performance to the same models trained using Backpropagation. We use following performance metrics for the comparison: (i) Training error; (ii) Validation error; (iii) Memory usage; and (iv) Training time. Our results indicate that CoNNTrA models use 32x less memory and have errors at par with the Backpropagation models.