Geometric structure of shallow neural networks and constructive ${\mathcal L}^2$ cost minimization

TL;DR

This paper analyzes the geometric structure of shallow ReLU neural networks and provides explicit constructions for cost minimization without gradient descent, revealing bounds and local minima related to data structure.

Contribution

It introduces a geometric approach to construct upper bounds for cost minimization in shallow networks, explicitly characterizes local minima, and connects these to data structure without relying on gradient methods.

Findings

Upper bound on cost minimization of order O(δ_P)

Explicit local minimum for the case M=Q

Constructive network training that captures a Q-dimensional subspace

Abstract

In this paper, we approach the problem of cost (loss) minimization in underparametrized shallow ReLU networks through the explicit construction of upper bounds which appeal to the structure of classification data, without use of gradient descent. A key focus is on elucidating the geometric structure of approximate and precise minimizers. We consider an $L^2$ cost function, input space $\mathbb{R}^M$, output space ${\mathbb R}^Q$ with $Q\leq M$, and training input sample size that can be arbitrarily large. We prove an upper bound on the minimum of the cost function of order $O(\delta_P)$ where $\delta_P$ measures the signal-to-noise ratio of training data. In the special case $M=Q$, we explicitly determine an exact degenerate local minimum of the cost function, and show that the sharp value differs from the upper bound obtained for $Q\leq M$ by a relative error $O(\delta_P^2)$. The proof of the upper bound yields a constructively trained network; we show that it metrizes a particular $Q$-dimensional subspace in the input space ${\mathbb R}^M$. We comment on the characterization of the global minimum of the cost function in the given context.