DCT Perceptron Layer: A Transform Domain Approach for Convolution Layer

TL;DR

This paper introduces a DCT-based neural network layer called DCT-perceptron, replacing traditional convolution layers in ResNet, reducing parameters and computations while maintaining accuracy, and offering location-specific processing in the transform domain.

Contribution

The paper presents a novel DCT-perceptron layer that performs convolution in the transform domain, significantly reducing parameters and computations in ResNet architectures.

Findings

DCT-perceptron achieves comparable accuracy to ResNet with fewer parameters.

The layer reduces computational complexity and parameter count.

Inserting DCT-perceptron improves classification accuracy when combined with batch normalization.

Abstract

In this paper, we propose a novel Discrete Cosine Transform (DCT)-based neural network layer which we call DCT-perceptron to replace the $3\times3$ Conv2D layers in the Residual neural Network (ResNet). Convolutional filtering operations are performed in the DCT domain using element-wise multiplications by taking advantage of the Fourier and DCT Convolution theorems. A trainable soft-thresholding layer is used as the nonlinearity in the DCT perceptron. Compared to ResNet's Conv2D layer which is spatial-agnostic and channel-specific, the proposed layer is location-specific and channel-specific. The DCT-perceptron layer reduces the number of parameters and multiplications significantly while maintaining comparable accuracy results of regular ResNets in CIFAR-10 and ImageNet-1K. Moreover, the DCT-perceptron layer can be inserted with a batch normalization layer before the global average pooling layer in the conventional ResNets as an additional layer to improve classification accuracy.