Skip Connections Eliminate Singularities

TL;DR

This paper proposes that skip connections improve training of deep networks by eliminating singularities in the loss landscape caused by node permutation, elimination, and linear dependence, thus facilitating learning.

Contribution

It introduces a novel explanation that skip connections break certain singularities, supported by theoretical analysis and experiments on real-world datasets.

Findings

Skip connections break permutation symmetry of nodes.

They reduce the likelihood of node elimination.

They lessen linear dependence among nodes.

Abstract

Skip connections made the training of very deep networks possible and have become an indispensable component in a variety of neural architectures. A completely satisfactory explanation for their success remains elusive. Here, we present a novel explanation for the benefits of skip connections in training very deep networks. The difficulty of training deep networks is partly due to the singularities caused by the non-identifiability of the model. Several such singularities have been identified in previous works: (i) overlap singularities caused by the permutation symmetry of nodes in a given layer, (ii) elimination singularities corresponding to the elimination, i.e. consistent deactivation, of nodes, (iii) singularities generated by the linear dependence of the nodes. These singularities cause degenerate manifolds in the loss landscape that slow down learning. We argue that skip connections eliminate these singularities by breaking the permutation symmetry of nodes, by reducing the possibility of node elimination and by making the nodes less linearly dependent. Moreover, for typical initializations, skip connections move the network away from the "ghosts" of these singularities and sculpt the landscape around them to alleviate the learning slow-down. These hypotheses are supported by evidence from simplified models, as well as from experiments with deep networks trained on real-world datasets.