No one-hidden-layer neural network can represent multivariable functions

TL;DR

This paper demonstrates that no single-hidden-layer neural network with ReLU activation can exactly represent certain multivariable functions, highlighting fundamental limitations in neural network expressiveness.

Contribution

It introduces parameter constraints for one-hidden-layer ReLU networks and proves the existence of smooth functions they cannot precisely model.

Findings

Constraints reduce network degrees of freedom

Existence of smooth functions not representable by single-hidden-layer networks

Theoretical limits on neural network expressiveness

Abstract

In a function approximation with a neural network, an input dataset is mapped to an output index by optimizing the parameters of each hidden-layer unit. For a unary function, we present constraints on the parameters and its second derivative by constructing a continuum version of a one-hidden-layer neural network with the rectified linear unit (ReLU) activation function. The network is accurately implemented because the constraints decrease the degrees of freedom of the parameters. We also explain the existence of a smooth binary function that cannot be precisely represented by any such neural network.