On the Computational Efficiency of Training Neural Networks

TL;DR

This paper examines the computational complexity of training neural networks, highlighting both theoretical hardness results and practical algorithms that enable efficient training in modern settings.

Contribution

It offers new insights into the complexity of neural network training and introduces algorithms that are both provably efficient and practically effective.

Findings

Identification of computational hardness in training neural networks

Development of new algorithms for efficient training of certain neural network types

Bridging theoretical complexity with practical training methods

Abstract

It is well-known that neural networks are computationally hard to train. On the other hand, in practice, modern day neural networks are trained efficiently using SGD and a variety of tricks that include different activation functions (e.g. ReLU), over-specification (i.e., train networks which are larger than needed), and regularization. In this paper we revisit the computational complexity of training neural networks from a modern perspective. We provide both positive and negative results, some of them yield new provably efficient and practical algorithms for training certain types of neural networks.