A Survey on Methods and Theories of Quantized Neural Networks

TL;DR

This survey reviews various methods and theories of quantized neural networks, highlighting their role in reducing memory and energy consumption for deploying deep learning models on resource-constrained devices.

Contribution

It provides a comprehensive overview of quantization techniques, challenges, and trends in neural networks, offering insights into their theoretical foundations and practical applications.

Findings

Quantization significantly reduces model size and energy use.

Trade-offs exist between quantization level and model accuracy.

Current research addresses challenges in maintaining performance with low-bit quantization.

Abstract

Deep neural networks are the state-of-the-art methods for many real-world tasks, such as computer vision, natural language processing and speech recognition. For all its popularity, deep neural networks are also criticized for consuming a lot of memory and draining battery life of devices during training and inference. This makes it hard to deploy these models on mobile or embedded devices which have tight resource constraints. Quantization is recognized as one of the most effective approaches to satisfy the extreme memory requirements that deep neural network models demand. Instead of adopting 32-bit floating point format to represent weights, quantized representations store weights using more compact formats such as integers or even binary numbers. Despite a possible degradation in predictive performance, quantization provides a potential solution to greatly reduce the model size and the energy consumption. In this survey, we give a thorough review of different aspects of quantized neural networks. Current challenges and trends of quantized neural networks are also discussed.