Residual Networks: Lyapunov Stability and Convex Decomposition

TL;DR

This paper explains why residual networks avoid degradation with depth by analyzing Lyapunov stability and introduces a convex-decomposition architecture that enhances generalization and stability.

Contribution

It demonstrates Lyapunov stability as a key factor in residual networks' performance and proposes a novel architecture for function approximation via convex decomposition.

Findings

Residual networks maintain stability regardless of depth.

The convex-decomposition architecture effectively approximates complex functions.

Parameters that change little during training help prevent overfitting.

Abstract

While training error of most deep neural networks degrades as the depth of the network increases, residual networks appear to be an exception. We show that the main reason for this is the Lyapunov stability of the gradient descent algorithm: for an arbitrarily chosen step size, the equilibria of the gradient descent are most likely to remain stable for the parametrization of residual networks. We then present an architecture with a pair of residual networks to approximate a large class of functions by decomposing them into a convex and a concave part. Some parameters of this model are shown to change little during training, and this imperfect optimization prevents overfitting the data and leads to solutions with small Lipschitz constants, while providing clues about the generalization of other deep networks.