Strong stationarity for optimal control problems with non-smooth integral equation constraints: Application to continuous DNNs

TL;DR

This paper develops strong stationarity conditions for optimal control problems constrained by non-smooth integral equations, specifically addressing challenges posed by ReLU activations in continuous neural network models.

Contribution

It introduces a novel approach to derive strong stationarity conditions without smoothing non-differentiable functions, filling a gap in continuous neural network theory.

Findings

Derived strong stationarity conditions for non-smooth integral constraints.

Addressed the challenge of non-differentiability in ReLU-based neural networks.

Provided a theoretical foundation for optimal control in continuous DNNs.

Abstract

Motivated by the residual type neural networks (ResNet), this paper studies optimal control problems constrained by a non-smooth integral equation. Such non-smooth equations, for instance, arise in the continuous representation of fractional deep neural networks (DNNs). Here the underlying non-differentiable function is the ReLU or max function. The control enters in a nonlinear and multiplicative manner and we additionally impose control constraints. Because of the presence of the non-differentiable mapping, the application of standard adjoint calculus is excluded. We derive strong stationary conditions by relying on the limited differentiability properties of the non-smooth map. While traditional approaches smoothen the non-differentiable function, no such smoothness is retained in our final strong stationarity system. Thus, this work also closes a gap which currently exists in continuous neural networks with ReLU type activation function.