Depth-Adaptive Neural Networks from the Optimal Control viewpoint

TL;DR

This paper introduces an adaptive neural network method inspired by optimal control, which iteratively refines the network depth to better approximate continuous models, improving efficiency and accuracy.

Contribution

It proposes a novel iterative adaptive algorithm that progressively increases network depth, converging to the continuous optimal control formulation.

Findings

The adaptive method converges to the continuous problem under certain conditions.

It mitigates over-parameterization issues by adaptively increasing depth.

Numerical examples demonstrate improved performance with the adaptive approach.

Abstract

In recent years, deep learning has been connected with optimal control as a way to define a notion of a continuous underlying learning problem. In this view, neural networks can be interpreted as a discretization of a parametric Ordinary Differential Equation which, in the limit, defines a continuous-depth neural network. The learning task then consists in finding the best ODE parameters for the problem under consideration, and their number increases with the accuracy of the time discretization. Although important steps have been taken to realize the advantages of such continuous formulations, most current learning techniques fix a discretization (i.e. the number of layers is fixed). In this work, we propose an iterative adaptive algorithm where we progressively refine the time discretization (i.e. we increase the number of layers). Provided that certain tolerances are met across the iterations, we prove that the strategy converges to the underlying continuous problem. One salient advantage of such a shallow-to-deep approach is that it helps to benefit in practice from the higher approximation properties of deep networks by mitigating over-parametrization issues. The performance of the approach is illustrated in several numerical examples.