Deep ReLU Programming

TL;DR

This paper analyzes the structure of ReLU neural networks, introduces an extended Simplex algorithm for efficient optimization across affine regions, and demonstrates its application to neural network training with L1 loss.

Contribution

It presents a novel algorithm extending the Simplex method to ReLU networks, enabling efficient optimization across affine regions and L1 neural network training.

Findings

Extended Simplex algorithm can efficiently navigate ReLU affine regions.

The method applies to LAD regression as a special case.

First layer neural networks can be trained with guaranteed L1 loss decrease.

Abstract

Feed-forward ReLU neural networks partition their input domain into finitely many "affine regions" of constant neuron activation pattern and affine behaviour. We analyze their mathematical structure and provide algorithmic primitives for an efficient application of linear programming related techniques for iterative minimization of such non-convex functions. In particular, we propose an extension of the Simplex algorithm which is iterating on induced vertices but, in addition, is able to change its feasible region computationally efficiently to adjacent "affine regions". This way, we obtain the Barrodale-Roberts algorithm for LAD regression as a special case, but also are able to train the first layer of neural networks with L1 training loss decreasing in every step.