Linear discriminant initialization for feed-forward neural networks

TL;DR

This paper introduces a novel initialization method for neural networks using linear discriminants, leading to faster training and higher accuracy, based on geometric insights into network structure.

Contribution

The paper proposes a linear discriminant-based initialization for the first layer of neural networks, improving training efficiency and accuracy.

Findings

Fewer training steps needed for convergence.

Higher asymptotic training accuracy.

Effective across different network architectures.

Abstract

Informed by the basic geometry underlying feed forward neural networks, we initialize the weights of the first layer of a neural network using the linear discriminants which best distinguish individual classes. Networks initialized in this way take fewer training steps to reach the same level of training, and asymptotically have higher accuracy on training data.