Efficient Higher-order Convolution for Small Kernels in Deep Learning

TL;DR

This paper introduces a new efficient method for higher-order Volterra filtering in deep convolutional neural networks, reducing memory and computation costs, and proposes an attention module that improves classification performance.

Contribution

The paper presents a novel low-cost higher-order Volterra filtering technique and a new attention module, HLA, for improved CNN performance.

Findings

Demonstrates computational efficiency over traditional Volterra filters.

Shows competitive classification improvements on CIFAR-100.

Provides source code for reproducibility.

Abstract

Deep convolutional neural networks (DCNNs) are a class of artificial neural networks, primarily for computer vision tasks such as segmentation and classification. Many nonlinear operations, such as activation functions and pooling strategies, are used in DCNNs to enhance their ability to process different signals with different tasks. Conceptional convolution, a linear filter, is the essential component of DCNNs while nonlinear convolution is generally implemented as higher-order Volterra filters, However, for Volterra filtering, significant memory and computational costs pose a primary limitation for its widespread application in DCNN applications. In this study, we propose a novel method to perform higher-order Volterra filtering with lower memory and computation cost in forward and backward pass in DCNN training. The proposed method demonstrates computational advantages compared with conventional Volterra filter implementation. Furthermore, based on the proposed method, a new attention module called Higher-order Local Attention Block (HLA) is proposed and tested on CIFAR-100 dataset, which shows competitive improvement for classification task. Source code is available at: https://github.com/WinterWen666/Efficient-High-Order-Volterra-Convolution.git