Towards an Optimal Control Perspective of ResNet Training

TL;DR

This paper formulates ResNet training as an optimal control problem, leading to a dynamic that encourages pruning of unnecessary layers, potentially improving efficiency and understanding of residual networks.

Contribution

It introduces an optimal control-based training formulation for ResNets, linking control theory with neural network training and suggesting a new layer pruning strategy.

Findings

Biases weights of unnecessary residual layers to vanish

Provides a theoretical framework connecting control theory and ResNet training

Suggests a new approach for layer pruning based on optimal control principles

Abstract

We propose a training formulation for ResNets reflecting an optimal control problem that is applicable for standard architectures and general loss functions. We suggest bridging both worlds via penalizing intermediate outputs of hidden states corresponding to stage cost terms in optimal control. For standard ResNets, we obtain intermediate outputs by propagating the state through the subsequent skip connections and the output layer. We demonstrate that our training dynamic biases the weights of the unnecessary deeper residual layers to vanish. This indicates the potential for a theory-grounded layer pruning strategy.