Convergence Analysis of Deep Residual Networks

TL;DR

This paper provides a mathematical analysis of the convergence behavior of deep Residual Networks as their depth increases, offering insights into their design and theoretical foundations.

Contribution

It introduces a matrix-vector framework for analyzing ResNets and establishes sufficient conditions for their convergence as depth tends to infinity.

Findings

Established a convergence criterion for deep ResNets

Provided a mathematical justification for ResNet design

Verified theoretical results with experiments on benchmark data

Abstract

Various powerful deep neural network architectures have made great contribution to the exciting successes of deep learning in the past two decades. Among them, deep Residual Networks (ResNets) are of particular importance because they demonstrated great usefulness in computer vision by winning the first place in many deep learning competitions. Also, ResNets were the first class of neural networks in the development history of deep learning that are really deep. It is of mathematical interest and practical meaning to understand the convergence of deep ResNets. We aim at characterizing the convergence of deep ResNets as the depth tends to infinity in terms of the parameters of the networks. Toward this purpose, we first give a matrix-vector description of general deep neural networks with shortcut connections and formulate an explicit expression for the networks by using the notions of activation domains and activation matrices. The convergence is then reduced to the convergence of two series involving infinite products of non-square matrices. By studying the two series, we establish a sufficient condition for pointwise convergence of ResNets. Our result is able to give justification for the design of ResNets. We also conduct experiments on benchmark machine learning data to verify our results.