The Computational Complexity of Training ReLU(s)

TL;DR

This paper proves that training depth-2 ReLU neural networks is NP-hard even for simple cases, but also provides a proper learning algorithm with exponential dependence on the number of units and error margin.

Contribution

It establishes the NP-hardness of training shallow ReLU networks and introduces a proper learning algorithm with specific complexity bounds.

Findings

Training ReLU networks is NP-hard even for a single ReLU.

Error minimization for two ReLUs is NP-hard in the realizable case.

Proper learning of depth-2 ReLUs is achievable with a specific exponential time complexity.

Abstract

We consider the computational complexity of training depth-2 neural networks composed of rectified linear units (ReLUs). We show that, even for the case of a single ReLU, finding a set of weights that minimizes the squared error (even approximately) for a given training set is NP-hard. We also show that for a simple network consisting of two ReLUs, the error minimization problem is NP-hard, even in the realizable case. We complement these hardness results by showing that, when the weights and samples belong to the unit ball, one can (agnostically) properly and reliably learn depth-2 ReLUs with $k$ units and error at most $\epsilon$ in time $2^{(k/\epsilon)^{O(1)}}n^{O(1)}$; this extends upon a previous work of Goel, Kanade, Klivans and Thaler (2017) which provided efficient improper learning algorithms for ReLUs.