Comparison of different convolutional neural network activation functions and methods for building ensembles

TL;DR

This paper investigates the performance of CNN ensembles built with various activation functions, including six new ones, across multiple biomedical datasets, demonstrating improved results over standard methods.

Contribution

It introduces six novel activation functions for CNNs and evaluates their effectiveness in ensemble models across diverse biomedical classification tasks.

Findings

Ensembles with diverse activation functions outperform single models.

New activation functions like 2D Mexican ReLU and TanELU show superior performance.

The approach is validated on Vgg16 and ResNet50 architectures.

Abstract

Recently, much attention has been devoted to finding highly efficient and powerful activation functions for CNN layers. Because activation functions inject different nonlinearities between layers that affect performance, varying them is one method for building robust ensembles of CNNs. The objective of this study is to examine the performance of CNN ensembles made with different activation functions, including six new ones presented here: 2D Mexican ReLU, TanELU, MeLU+GaLU, Symmetric MeLU, Symmetric GaLU, and Flexible MeLU. The highest performing ensemble was built with CNNs having different activation layers that randomly replaced the standard ReLU. A comprehensive evaluation of the proposed approach was conducted across fifteen biomedical data sets representing various classification tasks. The proposed method was tested on two basic CNN architectures: Vgg16 and ResNet50. Results demonstrate the superiority in performance of this approach. The MATLAB source code for this study will be available at https://github.com/LorisNanni.