Fast and Slow Gradient Approximation for Binary Neural Network Optimization

TL;DR

This paper introduces a novel gradient approximation method for Binary Neural Networks that leverages historical gradient data and layer-specific embeddings, resulting in faster convergence and improved accuracy.

Contribution

It proposes the Fast and Slow Gradient Generation (FSG) method with Historical Gradient Storage and Layer Recognition Embeddings to enhance hypernetwork-based BNN optimization.

Findings

FSG achieves faster convergence on CIFAR datasets.

The method reduces loss values compared to baselines.

Incorporating historical gradients improves gradient estimation accuracy.

Abstract

Binary Neural Networks (BNNs) have garnered significant attention due to their immense potential for deployment on edge devices. However, the non-differentiability of the quantization function poses a challenge for the optimization of BNNs, as its derivative cannot be backpropagated. To address this issue, hypernetwork based methods, which utilize neural networks to learn the gradients of non-differentiable quantization functions, have emerged as a promising approach due to their adaptive learning capabilities to reduce estimation errors. However, existing hypernetwork based methods typically rely solely on current gradient information, neglecting the influence of historical gradients. This oversight can lead to accumulated gradient errors when calculating gradient momentum during optimization. To incorporate historical gradient information, we design a Historical Gradient Storage (HGS) module, which models the historical gradient sequence to generate the first-order momentum required for optimization. To further enhance gradient generation in hypernetworks, we propose a Fast and Slow Gradient Generation (FSG) method. Additionally, to produce more precise gradients, we introduce Layer Recognition Embeddings (LRE) into the hypernetwork, facilitating the generation of layer-specific fine gradients. Extensive comparative experiments on the CIFAR-10 and CIFAR-100 datasets demonstrate that our method achieves faster convergence and lower loss values, outperforming existing baselines.Code is available at http://github.com/two-tiger/FSG .