On the Hardness of Learning One Hidden Layer Neural Networks

TL;DR

This paper demonstrates that learning one hidden layer ReLU neural networks with Gaussian inputs and noise is computationally hard under standard cryptographic assumptions, linking it to the hardness of the CLWE problem.

Contribution

It establishes the first formal hardness result for learning shallow neural networks under standard cryptographic assumptions.

Findings

Learning one hidden layer ReLU networks is hard under cryptographic assumptions.

Hardness holds even with polynomial size networks and Gaussian input/noise.

Connects neural network learning difficulty to the hardness of the CLWE problem.

Abstract

In this work, we consider the problem of learning one hidden layer ReLU neural networks with inputs from $\mathbb{R}^d$. We show that this learning problem is hard under standard cryptographic assumptions even when: (1) the size of the neural network is polynomial in $d$, (2) its input distribution is a standard Gaussian, and (3) the noise is Gaussian and polynomially small in $d$. Our hardness result is based on the hardness of the Continuous Learning with Errors (CLWE) problem, and in particular, is based on the largely believed worst-case hardness of approximately solving the shortest vector problem up to a multiplicative polynomial factor.