An Embedding of ReLU Networks and an Analysis of their Identifiability

TL;DR

This paper introduces an embedding for ReLU neural networks that is invariant to scalings and permutations, enabling analysis of their local identifiability from finite sample realizations.

Contribution

The paper proposes a novel invariant embedding for ReLU networks and derives conditions for their local identifiability from finite data samples.

Findings

Embedding is invariant to scalings and permutations.

Conditions for local identifiability are established.

Identifiability criteria are characterized for shallow networks.

Abstract

Neural networks with the Rectified Linear Unit (ReLU) nonlinearity are described by a vector of parameters $\theta$, and realized as a piecewise linear continuous function $R_{\theta}: x \in \mathbb R^{d} \mapsto R_{\theta}(x) \in \mathbb R^{k}$. Natural scalings and permutations operations on the parameters $\theta$ leave the realization unchanged, leading to equivalence classes of parameters that yield the same realization. These considerations in turn lead to the notion of identifiability -- the ability to recover (the equivalence class of) $\theta$ from the sole knowledge of its realization $R_{\theta}$. The overall objective of this paper is to introduce an embedding for ReLU neural networks of any depth, $\Phi(\theta)$, that is invariant to scalings and that provides a locally linear parameterization of the realization of the network. Leveraging these two key properties, we derive some conditions under which a deep ReLU network is indeed locally identifiable from the knowledge of the realization on a finite set of samples $x_{i} \in \mathbb R^{d}$. We study the shallow case in more depth, establishing necessary and sufficient conditions for the network to be identifiable from a bounded subset $\mathcal X \subseteq \mathbb R^{d}$.