Accelerating Inference of Networks in the Frequency Domain

TL;DR

This paper introduces a novel method for performing neural network inference directly in the frequency domain, significantly accelerating computation for sparse frequency networks while maintaining accuracy.

Contribution

It proposes a frequency domain inference chain that reduces the need for multiple frequency transforms, enabling faster inference with minimal accuracy loss.

Findings

Achieves over 100x speedup in inference time.

Maintains high accuracy despite significant speedup.

Requires only two frequency transforms per network, at start and end.

Abstract

It has been demonstrated that networks' parameters can be significantly reduced in the frequency domain with a very small decrease in accuracy. However, given the cost of frequency transforms, the computational complexity is not significantly decreased. In this work, we propose performing network inference in the frequency domain to speed up networks whose frequency parameters are sparse. In particular, we propose a frequency inference chain that is dual to the network inference in the spatial domain. In order to handle the non-linear layers, we make a compromise to apply non-linear operations on frequency data directly, which works effectively. Enabled by the frequency inference chain and the strategy for non-linear layers, the proposed approach completes the entire inference in the frequency domain. Unlike previous approaches which require extra frequency or inverse transforms for all layers, the proposed approach only needs the frequency transform and its inverse once at the beginning and once at the end of a network. Comparisons with state-of-the-art methods demonstrate that the proposed approach significantly improves accuracy in the case of a high speedup ratio (over 100x). The source code is available at \url{https://github.com/guanfangdong/FreqNet-Infer}.