On the inductive bias of infinite-depth ResNets and the bottleneck rank

TL;DR

This paper analyzes the inductive bias of deep ResNets, revealing that they tend to favor low bottleneck rank solutions, which influences their generalization and expressivity.

Contribution

It provides a theoretical characterization of the inductive bias of infinite-depth ResNets, connecting it to nuclear norm and rank minimization.

Findings

Deep linear ResNets minimize a combination of nuclear norm and rank.

Deep nonlinear ResNets are biased towards low bottleneck rank solutions.

The inductive bias can be controlled via hyperparameters.

Abstract

We compute the minimum-norm weights of a deep linear ResNet, and find that the inductive bias of this architecture lies between minimizing nuclear norm and rank. This implies that, with appropriate hyperparameters, deep nonlinear ResNets have an inductive bias towards minimizing bottleneck rank.