Efficient Algorithms for Learning Depth-2 Neural Networks with General ReLU Activations

TL;DR

This paper introduces polynomial time algorithms for efficiently learning depth-2 neural networks with ReLU activations, including bias terms, by tensor decomposition and Hermite expansion techniques.

Contribution

It provides the first polynomial time algorithms for learning ReLU networks with biases, using tensor decomposition and Hermite expansion methods.

Findings

Algorithms are polynomial time and sample efficient.

Successfully handle networks with bias terms.

Establish identifiability of network parameters.

Abstract

We present polynomial time and sample efficient algorithms for learning an unknown depth-2 feedforward neural network with general ReLU activations, under mild non-degeneracy assumptions. In particular, we consider learning an unknown network of the form $f(x) = {a}^{\mathsf{T}}\sigma({W}^\mathsf{T}x+b)$, where $x$ is drawn from the Gaussian distribution, and $\sigma(t) := \max(t,0)$ is the ReLU activation. Prior works for learning networks with ReLU activations assume that the bias $b$ is zero. In order to deal with the presence of the bias terms, our proposed algorithm consists of robustly decomposing multiple higher order tensors arising from the Hermite expansion of the function $f(x)$. Using these ideas we also establish identifiability of the network parameters under minimal assumptions.