Injectivity of ReLU-layers: Tools from Frame Theory

TL;DR

This paper analyzes the injectivity of ReLU neural network layers using frame theory, providing conditions, methods for approximation, and explicit reconstruction formulas to understand information preservation.

Contribution

It introduces a frame theoretic approach to characterize ReLU layer injectivity, including practical methods for approximation and explicit reconstruction formulas.

Findings

Provides sufficient conditions for ReLU layer injectivity on bounded domains

Develops methods to numerically approximate maximal bias for given weights and data domains

Derives explicit reconstruction formulas based on frame theory duality

Abstract

Injectivity is the defining property of a mapping that ensures no information is lost and any input can be perfectly reconstructed from its output. By performing hard thresholding, the ReLU function naturally interferes with this property, making the injectivity analysis of ReLU layers in neural networks a challenging yet intriguing task that has not yet been fully solved. This article establishes a frame theoretic perspective to approach this problem. The main objective is to develop a comprehensive characterization of the injectivity behavior of ReLU layers in terms of all three involved ingredients: (i) the weights, (ii) the bias, and (iii) the domain where the data is drawn from. Maintaining a focus on practical applications, we limit our attention to bounded domains and present two methods for numerically approximating a maximal bias for given weights and data domains. These methods provide sufficient conditions for the injectivity of a ReLU layer on those domains and yield a novel practical methodology for studying the information loss in ReLU layers. Finally, we derive explicit reconstruction formulas based on the duality concept from frame theory.