A case where a spindly two-layer linear network whips any neural network with a fully connected input layer

TL;DR

The paper demonstrates that a simple two-layer linear network can efficiently learn sparse targets from Hadamard matrix data, outperforming fully connected networks that struggle with such tasks when trained with gradient descent.

Contribution

It proves that sparse input layers enable efficient learning of sparse targets with gradient descent, contrasting with the limitations of fully connected networks.

Findings

Two-layer linear network achieves expected square loss of log d / k.

Fully connected networks have expected loss of 1 - k/(d-1).

Sparse input layers are essential for efficient learning of sparse targets.

Abstract

It was conjectured that any neural network of any structure and arbitrary differentiable transfer functions at the nodes cannot learn the following problem sample efficiently when trained with gradient descent: The instances are the rows of a $d$-dimensional Hadamard matrix and the target is one of the features, i.e. very sparse. We essentially prove this conjecture: We show that after receiving a random training set of size $k < d$, the expected square loss is still $1-\frac{k}{(d-1)}$. The only requirement needed is that the input layer is fully connected and the initial weight vectors of the input nodes are chosen from a rotation invariant distribution. Surprisingly the same type of problem can be solved drastically more efficient by a simple 2-layer linear neural network in which the $d$ inputs are connected to the output node by chains of length 2 (Now the input layer has only one edge per input). When such a network is trained by gradient descent, then it has been shown that its expected square loss is $\frac{\log d}{k}$. Our lower bounds essentially show that a sparse input layer is needed to sample efficiently learn sparse targets with gradient descent when the number of examples is less than the number of input features.