# New pointwise convolution in Deep Neural Networks through Extremely Fast and Non Parametric Transforms

## TL;DR

This paper introduces a novel pointwise convolution method using conventional transforms like DWHT, significantly reducing computational complexity and parameters in neural networks while maintaining or improving accuracy.

## Contribution

It proposes applying traditional transforms to pointwise convolution in DNNs, especially DWHT, to enhance efficiency without accuracy loss.

## Key findings

- DWHT-based model achieved 1.49% accuracy increase.
- Parameters reduced by 79.1%.
- FLOPs reduced by 48.4%.

## Abstract

Some conventional transforms such as Discrete Walsh-Hadamard Transform (DWHT) and Discrete Cosine Transform (DCT) have been widely used as feature extractors in image processing but rarely applied in neural networks. However, we found that these conventional transforms have the ability to capture the cross-channel correlations without any learnable parameters in DNNs. This paper firstly proposes to apply conventional transforms to pointwise convolution, showing that such transforms significantly reduce the computational complexity of neural networks without accuracy performance degradation. Especially for DWHT, it requires no floating point multiplications but only additions and subtractions, which can considerably reduce computation overheads. In addition, its fast algorithm further reduces complexity of floating point addition from $\mathcal{O}(n^2)$ to $\mathcal{O}(n\log n)$. These nice properties construct extremely efficient networks in the number parameters and operations, enjoying accuracy gain. Our proposed DWHT-based model gained 1.49\% accuracy increase with 79.1\% reduced parameters and 48.4\% reduced FLOPs compared with its baseline model (MoblieNet-V1) on the CIFAR 100 dataset.