Scaling of learning time for high dimensional inputs

TL;DR

This paper provides a theoretical analysis showing that learning times in high-dimensional neural networks grow supralinearly with input dimension, highlighting fundamental limitations and guiding optimal network design.

Contribution

It introduces a geometric framework for analyzing how learning time scales with input dimensionality in Hebbian ICA models, revealing a supralinear growth pattern.

Findings

Learning times increase supralinearly with input dimension.

High-dimensional inputs lead to smaller gradients and longer learning times.

The results suggest fundamental limits for learning efficiency in high-dimensional spaces.

Abstract

Representation learning from complex data typically involves models with a large number of parameters, which in turn require large amounts of data samples. In neural network models, model complexity grows with the number of inputs to each neuron, with a trade-off between model expressivity and learning time. A precise characterization of this trade-off would help explain the connectivity and learning times observed in artificial and biological networks. We present a theoretical analysis of how learning time depends on input dimensionality for a Hebbian learning model performing independent component analysis. Based on the geometry of high-dimensional spaces, we show that the learning dynamics reduce to a unidimensional problem, with learning times dependent only on initial conditions. For higher input dimensions, initial parameters have smaller learning gradients and larger learning times. We find that learning times have supralinear scaling, becoming quickly prohibitive for high input dimensions. These results reveal a fundamental limitation for learning in high dimensions and help elucidate how the optimal design of neural networks depends on data complexity. Our approach outlines a new framework for analyzing learning dynamics and model complexity in neural network models.