Fast Walsh-Hadamard Transform and Smooth-Thresholding Based Binary Layers in Deep Neural Networks

TL;DR

This paper introduces a novel neural network layer using fast Walsh-Hadamard transform and smooth-thresholding to replace traditional convolution layers, significantly reducing parameters and increasing processing speed with minimal accuracy loss.

Contribution

The paper presents a new layer based on Walsh-Hadamard transform and smooth-thresholding, enabling multiplication-free operations and efficient parameter reduction in deep neural networks.

Findings

Reduces MobileNet-V2 parameters from 2.27M to 540K.

Processes data approximately twice as fast as standard 1x1 convolutions on NVIDIA Jetson Nano.

Maintains competitive accuracy with slight performance degradation.

Abstract

In this paper, we propose a novel layer based on fast Walsh-Hadamard transform (WHT) and smooth-thresholding to replace $1\times 1$ convolution layers in deep neural networks. In the WHT domain, we denoise the transform domain coefficients using the new smooth-thresholding non-linearity, a smoothed version of the well-known soft-thresholding operator. We also introduce a family of multiplication-free operators from the basic 2$\times$2 Hadamard transform to implement $3\times 3$ depthwise separable convolution layers. Using these two types of layers, we replace the bottleneck layers in MobileNet-V2 to reduce the network's number of parameters with a slight loss in accuracy. For example, by replacing the final third bottleneck layers, we reduce the number of parameters from 2.270M to 540K. This reduces the accuracy from 95.21\% to 92.98\% on the CIFAR-10 dataset. Our approach significantly improves the speed of data processing. The fast Walsh-Hadamard transform has a computational complexity of $O(m\log_2 m)$. As a result, it is computationally more efficient than the $1\times1$ convolution layer. The fast Walsh-Hadamard layer processes a tensor in $\mathbb{R}^{10\times32\times32\times1024}$ about 2 times faster than $1\times1$ convolution layer on NVIDIA Jetson Nano computer board.