Hyperspherical Loss-Aware Ternary Quantization

TL;DR

This paper introduces a hyperspherical loss-aware ternary quantization method that improves gradient accuracy and model performance in discrete space, enhancing the effectiveness of ternary neural network quantization.

Contribution

It proposes a novel regularization and re-scaling approach to improve gradient accuracy for ternary quantization, outperforming existing methods.

Findings

Significant accuracy improvements in image classification.

Enhanced object detection performance.

Effective gradient approximation via re-scaling.

Abstract

Most of the existing works use projection functions for ternary quantization in discrete space. Scaling factors and thresholds are used in some cases to improve the model accuracy. However, the gradients used for optimization are inaccurate and result in a notable accuracy gap between the full precision and ternary models. To get more accurate gradients, some works gradually increase the discrete portion of the full precision weights in the forward propagation pass, e.g., using temperature-based Sigmoid function. Instead of directly performing ternary quantization in discrete space, we push full precision weights close to ternary ones through regularization term prior to ternary quantization. In addition, inspired by the temperature-based method, we introduce a re-scaling factor to obtain more accurate gradients by simulating the derivatives of Sigmoid function. The experimental results show that our method can significantly improve the accuracy of ternary quantization in both image classification and object detection tasks.