Spectral Convolution Networks

TL;DR

This paper explores implementing both convolution and activation functions directly in the frequency domain using Fourier and Laplace transforms, enabling fully spectral convolutional neural networks with reduced computational complexity.

Contribution

It introduces spectral activation functions and an efficient algorithm for spectral convolution and activation, facilitating fully spectral CNNs.

Findings

Spectral activation functions can be implemented in the Fourier domain.

An efficient Laplace transform-based algorithm for spectral convolution and activation.

Potential for fully spectral CNNs with reduced transform computations.

Abstract

Previous research has shown that computation of convolution in the frequency domain provides a significant speedup versus traditional convolution network implementations. However, this performance increase comes at the expense of repeatedly computing the transform and its inverse in order to apply other network operations such as activation, pooling, and dropout. We show, mathematically, how convolution and activation can both be implemented in the frequency domain using either the Fourier or Laplace transformation. The main contributions are a description of spectral activation under the Fourier transform and a further description of an efficient algorithm for computing both convolution and activation under the Laplace transform. By computing both the convolution and activation functions in the frequency domain, we can reduce the number of transforms required, as well as reducing overall complexity. Our description of a spectral activation function, together with existing spectral analogs of other network functions may then be used to compose a fully spectral implementation of a convolution network.