Trainable Activation Function in Image Classification

TL;DR

This paper introduces trainable activation functions for deep neural networks, using Fourier series and linear combinations, demonstrating improved performance over ReLU on CIFAR-10.

Contribution

It proposes a novel method to make activation functions trainable via Fourier series and linear combinations, enhancing CNN performance.

Findings

Trainable activation functions outperform ReLU on CIFAR-10.

Fourier series-based activation improves CNN accuracy.

Optimization with PSO further enhances network performance.

Abstract

In the current research of neural networks, the activation function is manually specified by human and not able to change themselves during training. This paper focus on how to make the activation function trainable for deep neural networks. We use series and linear combination of different activation functions make activation functions continuously variable. Also, we test the performance of CNNs with Fourier series simulated activation(Fourier-CNN) and CNNs with linear combined activation function (LC-CNN) on Cifar-10 dataset. The result shows our trainable activation function reveals better performance than the most used ReLU activation function. Finally, we improves the performance of Fourier-CNN with Autoencoder, and test the performance of PSO algorithm in optimizing the parameters of networks