Evolution of Activation Functions for Deep Learning-Based Image Classification

TL;DR

This paper introduces a coevolutionary algorithm to optimize activation functions in neural networks, demonstrating its effectiveness across multiple datasets and outperforming other methods in finding suitable AFs and architectures.

Contribution

It proposes a novel three-population coevolutionary algorithm for evolving activation functions, improving the selection process for AFs in neural networks.

Findings

Coevolution outperforms other methods in finding effective AFs.

The algorithm is effective across multiple datasets.

It enhances neural network performance by optimizing AFs.

Abstract

Activation functions (AFs) play a pivotal role in the performance of neural networks. The Rectified Linear Unit (ReLU) is currently the most commonly used AF. Several replacements to ReLU have been suggested but improvements have proven inconsistent. Some AFs exhibit better performance for specific tasks, but it is hard to know a priori how to select the appropriate one(s). Studying both standard fully connected neural networks (FCNs) and convolutional neural networks (CNNs), we propose a novel, three-population, coevolutionary algorithm to evolve AFs, and compare it to four other methods, both evolutionary and non-evolutionary. Tested on four datasets -- MNIST, FashionMNIST, KMNIST, and USPS -- coevolution proves to be a performant algorithm for finding good AFs and AF architectures.