Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precision

TL;DR

This paper proposes a dual-level precision scheme for neural network inference that reduces resource demands by combining low and high precision levels, with targeted error correction to maintain accuracy, suitable for edge devices.

Contribution

It introduces neural precision polarization, a novel approach that separates precision levels for efficient inference and employs low-rank error correction paths to recover accuracy.

Findings

Achieves approximately 464 TOPS/Watt MAC efficiency.

Enables effective error recovery with low-rank paths.

Supports quantization to NF4 or INT8 for edge deployment.

Abstract

We introduce a precision polarization scheme for DNN inference that utilizes only very low and very high precision levels, assigning low precision to the majority of network weights and activations while reserving high precision paths for targeted error compensation. This separation allows for distinct optimization of each precision level, thereby reducing memory and computation demands without compromising model accuracy. In the discussed approach, a floating-point model can be trained in the cloud and then downloaded to an edge device, where network weights and activations are directly quantized to meet the edge devices' desired level, such as NF4 or INT8. To address accuracy loss from quantization, surrogate paths are introduced, leveraging low-rank approximations on a layer-by-layer basis. These paths are trained with a sensitivity-based metric on minimal training data to recover accuracy loss under quantization as well as due to process variability, such as when the main prediction path is implemented using analog acceleration. Our simulation results show that neural precision polarization enables approximately 464 TOPS per Watt MAC efficiency and reliability by integrating rank-8 error recovery paths with highly efficient, though potentially unreliable, bit plane-wise compute-in-memory processing.