Activation Functions: Do They Represent A Trade-Off Between Modular Nature of Neural Networks And Task Performance

TL;DR

This paper investigates whether ReLU activation functions optimize the balance between neural network modularity and task performance, questioning if alternative functions could enhance modularity without sacrificing efficiency.

Contribution

It explores the potential trade-offs between activation functions, particularly ReLU, in terms of modularity and overall neural network performance.

Findings

ReLU accelerates training convergence.

Trade-offs exist between modularity and performance.

Alternative activation functions may improve modularity.

Abstract

Current research suggests that the key factors in designing neural network architectures involve choosing number of filters for every convolution layer, number of hidden neurons for every fully connected layer, dropout and pruning. The default activation function in most cases is the ReLU, as it has empirically shown faster training convergence. We explore whether ReLU is the best choice if one is aiming to desire better modularity structure within a neural network.