An Analysis of State-of-the-art Activation Functions For Supervised Deep Neural Network

TL;DR

This paper analyzes various state-of-the-art activation functions in deep neural networks, evaluating their performance across different architectures and datasets for supervised classification tasks.

Contribution

It provides a comparative analysis of multiple activation functions using two different deep learning architectures and datasets.

Findings

ReLU performs well on MNIST with MLP.

ELU and SELU show advantages in certain architectures.

Activation function choice impacts performance across datasets.

Abstract

This paper provides an analysis of state-of-the-art activation functions with respect to supervised classification of deep neural network. These activation functions comprise of Rectified Linear Units (ReLU), Exponential Linear Unit (ELU), Scaled Exponential Linear Unit (SELU), Gaussian Error Linear Unit (GELU), and the Inverse Square Root Linear Unit (ISRLU). To evaluate, experiments over two deep learning network architectures integrating these activation functions are conducted. The first model, basing on Multilayer Perceptron (MLP), is evaluated with MNIST dataset to perform these activation functions. Meanwhile, the second model, likely VGGish-based architecture, is applied for Acoustic Scene Classification (ASC) Task 1A in DCASE 2018 challenge, thus evaluate whether these activation functions work well in different datasets as well as different network architectures.