Deep ReLU Programming
Peter Hinz, Sara van de Geer

TL;DR
This paper analyzes the structure of ReLU neural networks, introduces an extended Simplex algorithm for efficient optimization across affine regions, and demonstrates its application to neural network training with L1 loss.
Contribution
It presents a novel algorithm extending the Simplex method to ReLU networks, enabling efficient optimization across affine regions and L1 neural network training.
Findings
Extended Simplex algorithm can efficiently navigate ReLU affine regions.
The method applies to LAD regression as a special case.
First layer neural networks can be trained with guaranteed L1 loss decrease.
Abstract
Feed-forward ReLU neural networks partition their input domain into finitely many "affine regions" of constant neuron activation pattern and affine behaviour. We analyze their mathematical structure and provide algorithmic primitives for an efficient application of linear programming related techniques for iterative minimization of such non-convex functions. In particular, we propose an extension of the Simplex algorithm which is iterating on induced vertices but, in addition, is able to change its feasible region computationally efficiently to adjacent "affine regions". This way, we obtain the Barrodale-Roberts algorithm for LAD regression as a special case, but also are able to train the first layer of neural networks with L1 training loss decreasing in every step.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Sparse and Compressive Sensing Techniques · Machine Learning and ELM
