Review and Comparison of Commonly Used Activation Functions for Deep Neural Networks

TL;DR

This paper reviews and compares popular activation functions in deep neural networks, analyzing their properties, advantages, disadvantages, and suitable application scenarios to guide better selection.

Contribution

It provides a comprehensive comparison of common activation functions, highlighting their characteristics, limitations, and practical recommendations for neural network design.

Findings

ReLU is simple and efficient but can cause dying neurons.

Swish offers smooth activation with better performance in some cases.

Sigmoid functions can cause vanishing gradients.

Abstract

The primary neural networks decision-making units are activation functions. Moreover, they evaluate the output of networks neural node; thus, they are essential for the performance of the whole network. Hence, it is critical to choose the most appropriate activation function in neural networks calculation. Acharya et al. (2018) suggest that numerous recipes have been formulated over the years, though some of them are considered deprecated these days since they are unable to operate properly under some conditions. These functions have a variety of characteristics, which are deemed essential to successfully learning. Their monotonicity, individual derivatives, and finite of their range are some of these characteristics (Bach 2017). This research paper will evaluate the commonly used additive functions, such as swish, ReLU, Sigmoid, and so forth. This will be followed by their properties, own cons and pros, and particular formula application recommendations.