Convex Geometry of ReLU-layers, Injectivity on the Ball and Local Reconstruction

TL;DR

This paper investigates the injectivity of ReLU layers on a ball in Euclidean space using convex geometry and frame theory, providing conditions for invertibility and explicit reconstruction formulas.

Contribution

It introduces a new method to verify ReLU-layer injectivity based on convex geometry and bias constraints, with explicit reconstruction formulas.

Findings

A feasible method to verify ReLU-layer injectivity.

Explicit formulas for input reconstruction on the ball.

Quantification of ReLU-layer invertibility.

Abstract

The paper uses a frame-theoretic setting to study the injectivity of a ReLU-layer on the closed ball of $\mathbb{R}^n$ and its non-negative part. In particular, the interplay between the radius of the ball and the bias vector is emphasized. Together with a perspective from convex geometry, this leads to a computationally feasible method of verifying the injectivity of a ReLU-layer under reasonable restrictions in terms of an upper bound of the bias vector. Explicit reconstruction formulas are provided, inspired by the duality concept from frame theory. All this gives rise to the possibility of quantifying the invertibility of a ReLU-layer and a concrete reconstruction algorithm for any input vector on the ball.