Low Precision Neural Networks using Subband Decomposition

TL;DR

This paper introduces a method that decomposes images into frequency bands to enable neural networks to train with lower precision weights, significantly reducing model size and computational requirements while maintaining stability.

Contribution

It presents a novel approach combining subband decomposition with low-precision weights to improve training efficiency and model compression in deep neural networks.

Findings

Supports 2-4X reduction in weight precision

Achieves 17X reduction in DNN parameters

Enables more stable learning with lower precision

Abstract

Large-scale deep neural networks (DNN) have been successfully used in a number of tasks from image recognition to natural language processing. They are trained using large training sets on large models, making them computationally and memory intensive. As such, there is much interest in research development for faster training and test time. In this paper, we present a unique approach using lower precision weights for more efficient and faster training phase. We separate imagery into different frequency bands (e.g. with different information content) such that the neural net can better learn using less bits. We present this approach as a complement existing methods such as pruning network connections and encoding learning weights. We show results where this approach supports more stable learning with 2-4X reduction in precision with 17X reduction in DNN parameters.