Kernel function impact on convolutional neural networks

TL;DR

This paper explores the effects of different kernel functions in CNNs, proposing new methods like distortion-aware pooling and Kernelized Dense Layers to enhance performance and discriminative power, especially in fine-grained classification tasks.

Contribution

It introduces novel kernel-based layers and pooling techniques that improve CNN accuracy and detail sensitivity, outperforming classical methods on multiple datasets.

Findings

Kernel functions impact CNN performance significantly.

Distortion-aware pooling reduces overfitting effectively.

Kernelized Dense Layers enhance discriminative power.

Abstract

This paper investigates the usage of kernel functions at the different layers in a convolutional neural network. We carry out extensive studies of their impact on convolutional, pooling and fully-connected layers. We notice that the linear kernel may not be sufficiently effective to fit the input data distributions, whereas high order kernels prone to over-fitting. This leads to conclude that a trade-off between complexity and performance should be reached. We show how one can effectively leverage kernel functions, by introducing a more distortion aware pooling layers which reduces over-fitting while keeping track of the majority of the information fed into subsequent layers. We further propose Kernelized Dense Layers (KDL), which replace fully-connected layers, and capture higher order feature interactions. The experiments on conventional classification datasets i.e. MNIST, FASHION-MNIST and CIFAR-10, show that the proposed techniques improve the performance of the network compared to classical convolution, pooling and fully connected layers. Moreover, experiments on fine-grained classification i.e. facial expression databases, namely RAF-DB, FER2013 and ExpW demonstrate that the discriminative power of the network is boosted, since the proposed techniques improve the awareness to slight visual details and allows the network reaching state-of-the-art results.