Ensemble of Convolutional Neural Networks Trained with Different Activation Functions

TL;DR

This paper introduces an ensemble of CNNs trained with various activation functions, including a novel one, to enhance performance on small/medium biomedical datasets, outperforming standard ReLU-based models.

Contribution

It proposes a new ensemble method combining different activation functions, including a novel function, to improve CNN performance on biomedical datasets.

Findings

Ensemble outperforms individual activation functions with p-value 0.01.

The approach improves CNN accuracy on over 10 biomedical datasets.

The method is validated with Vgg16 and ResNet50 architectures.

Abstract

Activation functions play a vital role in the training of Convolutional Neural Networks. For this reason, to develop efficient and performing functions is a crucial problem in the deep learning community. Key to these approaches is to permit a reliable parameter learning, avoiding vanishing gradient problems. The goal of this work is to propose an ensemble of Convolutional Neural Networks trained using several different activation functions. Moreover, a novel activation function is here proposed for the first time. Our aim is to improve the performance of Convolutional Neural Networks in small/medium size biomedical datasets. Our results clearly show that the proposed ensemble outperforms Convolutional Neural Networks trained with standard ReLU as activation function. The proposed ensemble outperforms with a p-value of 0.01 each tested stand-alone activation function; for reliable performance comparison we have tested our approach in more than 10 datasets, using two well-known Convolutional Neural Network: Vgg16 and ResNet50. MATLAB code used here will be available at https://github.com/LorisNanni.