Towards an Understanding of Residual Networks Using Neural Tangent Hierarchy (NTH)

TL;DR

This paper analyzes the training dynamics of finite-width deep residual networks using the neural tangent hierarchy, revealing how skip connections contribute to their success over fully-connected networks.

Contribution

It extends neural tangent kernel analysis to ResNets with finite width, showing reduced width requirements and highlighting the importance of skip connections.

Findings

ResNets with smooth activation functions require cubic width in samples for analysis.

Skip connections are key to ResNet's superior performance.

Analysis suggests ResNet's structure is crucial for its training efficiency.

Abstract

Gradient descent yields zero training loss in polynomial time for deep neural networks despite non-convex nature of the objective function. The behavior of network in the infinite width limit trained by gradient descent can be described by the Neural Tangent Kernel (NTK) introduced in \cite{Jacot2018Neural}. In this paper, we study dynamics of the NTK for finite width Deep Residual Network (ResNet) using the neural tangent hierarchy (NTH) proposed in \cite{Huang2019Dynamics}. For a ResNet with smooth and Lipschitz activation function, we reduce the requirement on the layer width $m$ with respect to the number of training samples $n$ from quartic to cubic. Our analysis suggests strongly that the particular skip-connection structure of ResNet is the main reason for its triumph over fully-connected network.