On the existence of minimizers in shallow residual ReLU neural network optimization landscapes

TL;DR

This paper proves the existence of minimizers in the loss landscape of shallow residual ReLU neural networks, contrasting prior results that showed non-existence for smooth activations, by extending the search space to include discontinuous functions.

Contribution

It establishes the existence of minimizers for residual ReLU networks in settings where previous work found none, using an extended function space approach.

Findings

Minimizers exist in the loss landscape for residual ReLU networks.

Extended function space includes discontinuous responses, but minimizers are in the original class.

Contrasts with prior results showing non-existence for smooth activations.

Abstract

In this article, we show existence of minimizers in the loss landscape for residual artificial neural networks (ANNs) with multi-dimensional input layer and one hidden layer with ReLU activation. Our work contrasts earlier results in [D. Gallon, A. Jentzen, and F. Lindner, preprint, arXiv:2211.15641, 2022] and [P. Petersen, M. Raslan, and F. Voigtlaender, Found. Comput. Math., 21 (2021), pp. 375-444] which showed that in many situations minimizers do not exist for common smooth activation functions even in the case where the target functions are polynomials. The proof of the existence property makes use of a closure of the search space containing all functions generated by ANNs and additional discontinuous generalized responses. As we will show, the additional generalized responses in this larger space are suboptimal so that the minimum is attained in the original function class.