Optimal lower Lipschitz bounds for ReLU layers, saturation, and phase retrieval

TL;DR

This paper provides a unified theoretical framework for analyzing the injectivity and stability of ReLU neural network layers, saturation effects, and phase retrieval, deriving optimal lower Lipschitz bounds for these problems.

Contribution

It introduces a unified approach to three problems, deriving and establishing optimal lower Lipschitz bounds for ReLU layers, saturation, and phase retrieval.

Findings

Derived lower Lipschitz bounds for ReLU layers and saturation.

Bounds are optimal up to a constant factor.

Unified characterization of three related problems.

Abstract

The injectivity of ReLU layers in neural networks, the recovery of vectors from clipped or saturated measurements, and (real) phase retrieval in $\mathbb{R}^n$ allow for a similar problem formulation and characterization using frame theory. In this paper, we revisit all three problems with a unified perspective and derive lower Lipschitz bounds for ReLU layers and clipping which are analogous to the previously known result for phase retrieval and are optimal up to a constant factor.