The Quest for the Golden Activation Function

TL;DR

This paper emphasizes the importance of activation functions in deep neural networks, demonstrating that task-specific learned functions via genetic algorithms can outperform standard choices, and introduces two new activation functions.

Contribution

It introduces a genetic algorithm-based framework to automatically learn optimal activation functions for specific tasks and proposes two novel activation functions, ELiSH and HardELiSH.

Findings

Learned activation functions vary across tasks.

Proposed methods outperform standard baselines.

New activation functions improve classification accuracy.

Abstract

Deep Neural Networks have been shown to be beneficial for a variety of tasks, in particular allowing for end-to-end learning and reducing the requirement for manual design decisions. However, still many parameters have to be chosen in advance, also raising the need to optimize them. One important, but often ignored system parameter is the selection of a proper activation function. Thus, in this paper we target to demonstrate the importance of activation functions in general and show that for different tasks different activation functions might be meaningful. To avoid the manual design or selection of activation functions, we build on the idea of genetic algorithms to learn the best activation function for a given task. In addition, we introduce two new activation functions, ELiSH and HardELiSH, which can easily be incorporated in our framework. In this way, we demonstrate for three different image classification benchmarks that different activation functions are learned, also showing improved results compared to typically used baselines.