Vanishing Nodes: Another Phenomenon That Makes Training Deep Neural Networks Difficult

TL;DR

This paper introduces the concept of vanishing nodes, a phenomenon where hidden nodes in deep neural networks become highly correlated, increasing training difficulty, and proposes a metric to measure this effect.

Contribution

The paper defines vanishing nodes, develops the vanishing node indicator (VNI), and demonstrates its impact on training deep networks, distinct from vanishing gradients.

Findings

Vanishing nodes increase with network depth and are inversely related to width.

When VNI reaches 1, the network's effective nodes reduce to one, hindering learning.

Deeper networks are more prone to training failure due to vanishing nodes.

Abstract

It is well known that the problem of vanishing/exploding gradients is a challenge when training deep networks. In this paper, we describe another phenomenon, called vanishing nodes, that also increases the difficulty of training deep neural networks. As the depth of a neural network increases, the network's hidden nodes have more highly correlated behavior. This results in great similarities between these nodes. The redundancy of hidden nodes thus increases as the network becomes deeper. We call this problem vanishing nodes, and we propose the metric vanishing node indicator (VNI) for quantitatively measuring the degree of vanishing nodes. The VNI can be characterized by the network parameters, which is shown analytically to be proportional to the depth of the network and inversely proportional to the network width. The theoretical results show that the effective number of nodes vanishes to one when the VNI increases to one (its maximal value), and that vanishing/exploding gradients and vanishing nodes are two different challenges that increase the difficulty of training deep neural networks. The numerical results from the experiments suggest that the degree of vanishing nodes will become more evident during back-propagation training, and that when the VNI is equal to 1, the network cannot learn simple tasks (e.g. the XOR problem) even when the gradients are neither vanishing nor exploding. We refer to this kind of gradients as the walking dead gradients, which cannot help the network converge when having a relatively large enough scale. Finally, the experiments show that the likelihood of failed training increases as the depth of the network increases. The training will become much more difficult due to the lack of network representation capability.