# Deep Representation with ReLU Neural Networks

## TL;DR

This paper analyzes deep ReLU neural networks from a signal processing perspective, describing their affine linear regions and atomic decompositions to better understand their representations and stability.

## Contribution

It provides a detailed description of the affine linear regions in ReLU networks and proposes conditions for stabilizing learning independent of network depth.

## Key findings

- Characterization of affine linear regions in ReLU networks
- Atomic decomposition of neural representations
- Conditions for learning stability

## Abstract

We consider deep feedforward neural networks with rectified linear units from a signal processing perspective. In this view, such representations mark the transition from using a single (data-driven) linear representation to utilizing a large collection of affine linear representations tailored to particular regions of the signal space. This paper provides a precise description of the individual affine linear representations and corresponding domain regions that the (data-driven) neural network associates to each signal of the input space. In particular, we describe atomic decompositions of the representations and, based on estimating their Lipschitz regularity, suggest some conditions that can stabilize learning independent of the network depth. Such an analysis may promote further theoretical insight from both the signal processing and machine learning communities.