Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers

TL;DR

This paper investigates adaptive precision training for neural networks, focusing on quantifying backpropagation with fixed-point numbers to improve training efficiency while maintaining accuracy.

Contribution

It introduces a novel approach to adaptive precision training that effectively quantifies backpropagation using fixed-point arithmetic, addressing accuracy loss issues.

Findings

Quantization impacts training accuracy significantly.

Adaptive precision methods can mitigate accuracy loss.

Fixed-point backpropagation is feasible with proper quantization.

Abstract

Adaptive Precision Training: Quantify Back Propagation in Neural Networks with Fixed-point Numbers. Recent emerged quantization technique has been applied to inference of deep neural networks for fast and efficient execution. However, directly applying quantization in training can cause significant accuracy loss, thus remaining an open challenge.