Universal Solutions of Feedforward ReLU Networks for Interpolations

TL;DR

This paper develops a comprehensive theoretical framework for understanding solutions of feedforward ReLU networks for interpolation, covering shallow and deep architectures, and introduces the sparse-matrix principle to explain deep network behaviors.

Contribution

It introduces the interpolation matrix framework for analyzing ReLU networks, classifies solutions, and presents the sparse-matrix principle for deep layers, advancing theoretical understanding.

Findings

Classification of solutions for three-layer networks

Introduction of the sparse-matrix principle for deep networks

Explanation of multi-output network mechanisms

Abstract

This paper provides a theoretical framework on the solution of feedforward ReLU networks for interpolations, in terms of what is called an interpolation matrix, which is the summary, extension and generalization of our three preceding works, with the expectation that the solution of engineering could be included in this framework and finally understood. To three-layer networks, we classify different kinds of solutions and model them in a normalized form; the solution finding is investigated by three dimensions, including data, networks and the training; the mechanism of a type of overparameterization solution is interpreted. To deep-layer networks, we present a general result called sparse-matrix principle, which could describe some basic behavior of deep layers and explain the phenomenon of the sparse-activation mode that appears in engineering applications associated with brain science; an advantage of deep layers compared to shallower ones is manifested in this principle. As applications, a general solution of deep neural networks for classifications is constructed by that principle; and we also use the principle to study the data-disentangling property of encoders. Analogous to the three-layer case, the solution of deep layers is also explored through several dimensions. The mechanism of multi-output neural networks is explained from the perspective of interpolation matrices.