Randomly Initialized One-Layer Neural Networks Make Data Linearly Separable

TL;DR

This paper proves that wide, randomly initialized one-layer neural networks can, with high probability, transform arbitrary data sets into linearly separable ones without training, offering theoretical insights into neural network capabilities.

Contribution

It establishes bounds on the width needed for random one-layer networks to achieve data separability, including a dimension-independent bound, advancing understanding of neural network initialization.

Findings

Randomly initialized networks can make data linearly separable without training.

Provided width bounds depend polynomially or are independent of input dimension.

Theoretical proof combines geometric and concentration of measure techniques.

Abstract

Recently, neural networks have demonstrated remarkable capabilities in mapping two arbitrary sets to two linearly separable sets. The prospect of achieving this with randomly initialized neural networks is particularly appealing due to the computational efficiency compared to fully trained networks. This paper contributes by establishing that, given sufficient width, a randomly initialized one-layer neural network can, with high probability, transform two sets into two linearly separable sets without any training. Moreover, we furnish precise bounds on the necessary width of the neural network for this phenomenon to occur. Our initial bound exhibits exponential dependence on the input dimension while maintaining polynomial dependence on all other parameters. In contrast, our second bound is independent of input dimension, effectively surmounting the curse of dimensionality. The main tools used in our proof heavily relies on a fusion of geometric principles and concentration of random matrices.