Theoretical Exploration of Solutions of Feedforward ReLU Networks

TL;DR

This paper provides a theoretical framework for understanding feedforward ReLU networks by analyzing their solutions as piecewise linear functions, offering insights into architecture components, overparameterization, and depth advantages.

Contribution

It introduces a universal solution framework for ReLU networks based on affine geometry, explaining architecture components, parameter sharing, and depth benefits.

Findings

Solutions for three-layer and deep networks are derived.

Interpretations of network components are provided.

Explanation of overparameterization via affine transforms.

Abstract

This paper aims to interpret the mechanism of feedforward ReLU networks by exploring their solutions for piecewise linear functions, through the deduction from basic rules. The constructed solution should be universal enough to explain some network architectures of engineering; in order for that, several ways are provided to enhance the solution universality. Some of the consequences of our theories include: Under affine-geometry background, the solutions of both three-layer networks and deep-layer networks are given, particularly for those architectures applied in practice, such as multilayer feedforward neural networks and decoders; We give clear and intuitive interpretations of each component of network architectures; The parameter-sharing mechanism for multi-outputs is investigated; We provide an explanation of overparameterization solutions in terms of affine transforms; Under our framework, an advantage of deep layers compared to shallower ones is natural to be obtained. Some intermediate results are the basic knowledge for the modeling or understanding of neural networks, such as the classification of data embedded in a higher-dimensional space, the generalization of affine transforms, the probabilistic model of matrix ranks, and the concept of distinguishable data sets.