Is the Skip Connection Provable to Reform the Neural Network Loss Landscape?

TL;DR

This paper provides a theoretical analysis showing that skip connections in deep ReLU neural networks improve the loss landscape's topology, making local minima shallower and potentially enhancing learning ability.

Contribution

It proves that skip connections inherit favorable properties from two-layer networks, controlling sub-level set connectedness and shallow local minima in deep networks.

Findings

Skip connections help control the connectedness of sub-level sets.

Local minima are at most O(m^{( heta-1)/n}) deep, shallower than in networks without skip connections.

Theoretical explanation for the effectiveness of skip connections in deep learning.

Abstract

The residual network is now one of the most effective structures in deep learning, which utilizes the skip connections to ``guarantee" the performance will not get worse. However, the non-convexity of the neural network makes it unclear whether the skip connections do provably improve the learning ability since the nonlinearity may create many local minima. In some previous works \cite{freeman2016topology}, it is shown that despite the non-convexity, the loss landscape of the two-layer ReLU network has good properties when the number $m$ of hidden nodes is very large. In this paper, we follow this line to study the topology (sub-level sets) of the loss landscape of deep ReLU neural networks with a skip connection and theoretically prove that the skip connection network inherits the good properties of the two-layer network and skip connections can help to control the connectedness of the sub-level sets, such that any local minima worse than the global minima of some two-layer ReLU network will be very ``shallow". The ``depth" of these local minima are at most $O(m^{(\eta-1)/n})$, where $n$ is the input dimension, $\eta<1$. This provides a theoretical explanation for the effectiveness of the skip connection in deep learning.