Regularization and Reparameterization Avoid Vanishing Gradients in Sigmoid-Type Networks

TL;DR

This paper investigates the vanishing-gradient problem in sigmoid-type networks, identifying its sources and proposing regularization and rescaling as effective remedies to improve training stability and revive sigmoid activations.

Contribution

It provides a mathematical analysis of the vanishing-gradient problem in sigmoid networks and introduces simple, effective solutions that enhance their training performance.

Findings

Regularization mitigates vanishing gradients caused by large parameters.

Rescaling helps address layer-wise gradient issues.

The remedies improve sigmoid network training in practice.

Abstract

Deep learning requires several design choices, such as the nodes' activation functions and the widths, types, and arrangements of the layers. One consideration when making these choices is the vanishing-gradient problem, which is the phenomenon of algorithms getting stuck at suboptimal points due to small gradients. In this paper, we revisit the vanishing-gradient problem in the context of sigmoid-type activation. We use mathematical arguments to highlight two different sources of the phenomenon, namely large individual parameters and effects across layers, and to illustrate two simple remedies, namely regularization and rescaling. We then demonstrate the effectiveness of the two remedies in practice. In view of the vanishing-gradient problem being a main reason why tanh and other sigmoid-type activation has become much less popular than relu-type activation, our results bring sigmoid-type activation back to the table.