PowerLinear Activation Functions with application to the first layer of CNNs

TL;DR

This paper introduces PowerLinear activation functions, particularly EvenPowLin, which enhance CNN performance by enabling better feature learning in the first layer and improve classification accuracy on inverted grayscale images.

Contribution

The paper proposes PowerLinear activation functions based on polynomial kernel generalization, allowing effective feature learning and improved classification in CNNs.

Findings

EvenPowLin functions are saturated neither in positive nor negative regions.

They activate negative inputs with the same magnitude as positive inputs.

CNNs with EvenPowLin classify inverted grayscale images as accurately as original images.

Abstract

Convolutional neural networks (CNNs) have become the state-of-the-art tool for dealing with unsolved problems in computer vision and image processing. Since the convolution operator is a linear operator, several generalizations have been proposed to improve the performance of CNNs. One way to increase the capability of the convolution operator is by applying activation functions on the inner product operator. In this paper, we will introduce PowerLinear activation functions, which are based on the polynomial kernel generalization of the convolution operator. EvenPowLin functions are the main branch of the PowerLinear activation functions. This class of activation functions is saturated neither in the positive input region nor in the negative one. Also, the negative inputs are activated with the same magnitude as the positive inputs. These features made the EvenPowLin activation functions able to be utilized in the first layer of CNN architectures and learn complex features of input images. Additionally, EvenPowLin activation functions are used in CNN models to classify the inversion of grayscale images as accurately as the original grayscale images, which is significantly better than commonly used activation functions.