A survey on recently proposed activation functions for Deep Learning

TL;DR

This survey reviews recent developments in activation functions for deep neural networks, discussing their properties, types, challenges, and potential solutions to improve neural network performance.

Contribution

It provides a comprehensive overview of various activation functions, highlighting recent proposals and analyzing their advantages and limitations in deep learning.

Findings

Summarizes key activation functions used in deep learning.

Identifies challenges and limitations of current activation functions.

Discusses alternative solutions and future directions.

Abstract

Artificial neural networks (ANN), typically referred to as neural networks, are a class of Machine Learning algorithms and have achieved widespread success, having been inspired by the biological structure of the human brain. Neural networks are inherently powerful due to their ability to learn complex function approximations from data. This generalization ability has been able to impact multidisciplinary areas involving image recognition, speech recognition, natural language processing, and others. Activation functions are a crucial sub-component of neural networks. They define the output of a node in the network given a set of inputs. This survey discusses the main concepts of activation functions in neural networks, including; a brief introduction to deep neural networks, a summary of what are activation functions and how they are used in neural networks, their most common properties, the different types of activation functions, some of the challenges, limitations, and alternative solutions faced by activation functions, concluding with the final remarks.