CBP: Backpropagation with constraint on weight precision using a pseudo-Lagrange multiplier method

TL;DR

This paper introduces CBP, a novel backpropagation algorithm that incorporates weight precision constraints using a pseudo-Lagrange multiplier, enabling effective training of deep neural networks with quantized weights.

Contribution

The proposed CBP algorithm uniquely integrates constraints into backpropagation via a pseudo-Lagrange multiplier, allowing constrained weight optimization without retraining from scratch.

Findings

CBP outperforms state-of-the-art methods on ImageNet with various constraints.

CBP achieves high accuracy with binary, ternary, and shift weight constraints.

The method effectively minimizes performance loss under diverse weight quantization constraints.

Abstract

Backward propagation of errors (backpropagation) is a method to minimize objective functions (e.g., loss functions) of deep neural networks by identifying optimal sets of weights and biases. Imposing constraints on weight precision is often required to alleviate prohibitive workloads on hardware. Despite the remarkable success of backpropagation, the algorithm itself is not capable of considering such constraints unless additional algorithms are applied simultaneously. To address this issue, we propose the constrained backpropagation (CBP) algorithm based on a pseudo-Lagrange multiplier method to obtain the optimal set of weights that satisfy a given set of constraints. The defining characteristic of the proposed CBP algorithm is the utilization of a Lagrangian function (loss function plus constraint function) as its objective function. We considered various types of constraints--binary, ternary, one-bit shift, and two-bit shift weight constraints. As a post-training method, CBP applied to AlexNet, ResNet-18, ResNet-50, and GoogLeNet on ImageNet, which were pre-trained using the conventional backpropagation. For all cases, the proposed algorithm outperforms the state-of-the-art methods on ImageNet, e.g., 66.6%, 74.4%, and 64.0% top-1 accuracy for ResNet-18, ResNet-50, and GoogLeNet with binary weights, respectively. This highlights CBP as a learning algorithm to address diverse constraints with the minimal performance loss by employing appropriate constraint functions.