Empirical Loss Landscape Analysis of Neural Network Activation Functions

TL;DR

This paper empirically analyzes how different activation functions affect the loss landscape of neural networks, revealing their influence on convexity, flatness, and generalization performance.

Contribution

It provides a comparative empirical study of loss landscapes for hyperbolic tangent, ReLU, and ELU activation functions, highlighting their distinct properties.

Findings

ReLU yields the most convex loss landscape.

ELU results in the least flat loss landscape and better generalization.

Wide and narrow valleys are present in all activation functions, with narrow valleys linked to saturation.

Abstract

Activation functions play a significant role in neural network design by enabling non-linearity. The choice of activation function was previously shown to influence the properties of the resulting loss landscape. Understanding the relationship between activation functions and loss landscape properties is important for neural architecture and training algorithm design. This study empirically investigates neural network loss landscapes associated with hyperbolic tangent, rectified linear unit, and exponential linear unit activation functions. Rectified linear unit is shown to yield the most convex loss landscape, and exponential linear unit is shown to yield the least flat loss landscape, and to exhibit superior generalisation performance. The presence of wide and narrow valleys in the loss landscape is established for all activation functions, and the narrow valleys are shown to correlate with saturated neurons and implicitly regularised network configurations.