On the algorithmic construction of deep ReLU networks

TL;DR

This paper explores the constructive, algorithmic perspective of deep ReLU networks, explicitly building networks for tasks like sorting, and analyzing their properties, recursion, and parallelism, highlighting their computational capabilities and limitations.

Contribution

It introduces a constructive approach to neural networks as algorithms, explicitly designing networks for specific tasks like sorting, and analyzing their structural properties and limitations.

Findings

Constructed neural networks can perform exact sorting with optimal complexity.

ReLU networks are typically recursive and parallel in structure.

Deep networks have advantages over shallow ones in recursion depth and capabilities.

Abstract

It is difficult to describe in mathematical terms what a neural network trained on data represents. On the other hand, there is a growing mathematical understanding of what neural networks are in principle capable of representing. Feedforward neural networks using the ReLU activation function represent continuous and piecewise linear functions and can approximate many others. The study of their expressivity addresses the question: which ones? Contributing to the available answers, we take the perspective of a neural network as an algorithm. In this analogy, a neural network is programmed constructively, rather than trained from data. An interesting example is a sorting algorithm: we explicitly construct a neural network that sorts its inputs exactly, not approximately, and that, in a sense, has optimal computational complexity if the input dimension is large. Such constructed networks may have several billion parameters. We construct and analyze several other examples, both existing and new. We find that, in these examples, neural networks as algorithms are typically recursive and parallel. Compared to conventional algorithms, ReLU networks are restricted by having to be continuous. Moreover, the depth of recursion is limited by the depth of the network, with deep networks having superior properties over shallow ones.