Band-limited Training and Inference for Convolutional Neural Networks

TL;DR

This paper investigates constraining the frequency spectra of convolutional filters during training, showing that CNNs are resilient to band-limiting which reduces resource usage without sacrificing accuracy.

Contribution

It introduces a band-limited training method for CNNs that controls resource consumption while maintaining high accuracy, without modifying existing architectures or algorithms.

Findings

Band-limited training reduces GPU and memory usage.

CNNs retain high accuracy with band-limited layers.

No changes to training algorithms or architectures are needed.

Abstract

The convolutional layers are core building blocks of neural network architectures. In general, a convolutional filter applies to the entire frequency spectrum of the input data. We explore artificially constraining the frequency spectra of these filters and data, called band-limiting, during training. The frequency domain constraints apply to both the feed-forward and back-propagation steps. Experimentally, we observe that Convolutional Neural Networks (CNNs) are resilient to this compression scheme and results suggest that CNNs learn to leverage lower-frequency components. In particular, we found: (1) band-limited training can effectively control the resource usage (GPU and memory); (2) models trained with band-limited layers retain high prediction accuracy; and (3) requires no modification to existing training algorithms or neural network architectures to use unlike other compression schemes.