Comparison of non-linear activation functions for deep neural networks on MNIST classification task

TL;DR

This paper compares various non-linear activation functions in deep neural networks on the MNIST classification task, analyzing their advantages, disadvantages, and the impact of weight initialization and network depth.

Contribution

It provides a comprehensive evaluation of different activation functions and the effects of weight initialization in deep neural networks for MNIST classification.

Findings

Activation functions significantly affect network performance.

Deeper networks improve classification accuracy.

Weight initialization impacts training stability.

Abstract

Activation functions play a key role in neural networks so it becomes fundamental to understand their advantages and disadvantages in order to achieve better performances. This paper will first introduce common types of non linear activation functions that are alternative to the well known sigmoid function and then evaluate their characteristics. Moreover deeper neural networks will be analysed because they positively influence the final performances compared to shallower networks. They also strictly depend on the weight initialisation hence the effect of drawing weights from Gaussian and uniform distribution will be analysed making particular attention on how the number of incoming and outgoing connection to a node influence the whole network.