Activation Function Optimization Scheme for Image Classification

TL;DR

This paper introduces an evolutionary optimization approach to discover new activation functions for image classification, resulting in the EELU functions that outperform existing options across various neural networks and datasets.

Contribution

The study presents a novel evolutionary framework for optimizing activation functions, leading to the development of the EELU functions that surpass current state-of-the-art functions in image classification.

Findings

EELU functions outperform standard activation functions in 92.8% of tested cases.

The optimization scheme successfully discovers activation functions better suited for diverse neural network architectures.

The best activation function identified is $-x ext{erf}(e^{-x})$, demonstrating the scheme's effectiveness.

Abstract

Activation function has a significant impact on the dynamics, convergence, and performance of deep neural networks. The search for a consistent and high-performing activation function has always been a pursuit during deep learning model development. Existing state-of-the-art activation functions are manually designed with human expertise except for Swish. Swish was developed using a reinforcement learning-based search strategy. In this study, we propose an evolutionary approach for optimizing activation functions specifically for image classification tasks, aiming to discover functions that outperform current state-of-the-art options. Through this optimization framework, we obtain a series of high-performing activation functions denoted as Exponential Error Linear Unit (EELU). The developed activation functions are evaluated for image classification tasks from two perspectives: (1) five state-of-the-art neural network architectures, such as ResNet50, AlexNet, VGG16, MobileNet, and Compact Convolutional Transformer which cover computationally heavy to light neural networks, and (2) eight standard datasets, including CIFAR10, Imagenette, MNIST, Fashion MNIST, Beans, Colorectal Histology, CottonWeedID15, and TinyImageNet which cover from typical machine vision benchmark, agricultural image applications to medical image applications. Finally, we statistically investigate the generalization of the resultant activation functions developed through the optimization scheme. With a Friedman test, we conclude that the optimization scheme is able to generate activation functions that outperform the existing standard ones in 92.8% cases among 28 different cases studied, and $-x\cdot erf(e^{-x})$ is found to be the best activation function for image classification generated by the optimization scheme.