On the Principles of ReLU Networks with One Hidden Layer

TL;DR

This paper investigates the interpretability of one-hidden-layer ReLU neural networks, providing theoretical and experimental insights into their solutions, which enhances understanding of their mechanisms and lays groundwork for analyzing deeper networks.

Contribution

It offers a systematic analysis of the solutions of one-hidden-layer ReLU networks, revealing their interpretability and approximation capabilities for both one-dimensional and higher-dimensional inputs.

Findings

Complete understanding of solutions for 1D inputs

Partial interpretability for higher-dimensional inputs

Advances towards revealing the black box of two-layer ReLU networks

Abstract

A neural network with one hidden layer or a two-layer network (regardless of the input layer) is the simplest feedforward neural network, whose mechanism may be the basis of more general network architectures. However, even to this type of simple architecture, it is also a ``black box''; that is, it remains unclear how to interpret the mechanism of its solutions obtained by the back-propagation algorithm and how to control the training process through a deterministic way. This paper systematically studies the first problem by constructing universal function-approximation solutions. It is shown that, both theoretically and experimentally, the training solution for the one-dimensional input could be completely understood, and that for a higher-dimensional input can also be well interpreted to some extent. Those results pave the way for thoroughly revealing the black box of two-layer ReLU networks and advance the understanding of deep ReLU networks.