Dense neural networks as sparse graphs and the lightning initialization

TL;DR

This paper presents the lightning initialization method for dense neural networks, which enhances information flow and accelerates training by ensuring complete input-output paths, especially effective at lower learning rates.

Contribution

It introduces a novel initialization technique that improves training efficiency and accuracy for dense networks by modeling them as sparse graphs with guaranteed information paths.

Findings

Lightning initialization improves training speed and accuracy.

Effectiveness is robust across different network complexities.

Lower learning rates enhance the positive impact of the initialization.

Abstract

Even though dense networks have lost importance today, they are still used as final logic elements. It could be shown that these dense networks can be simplified by the sparse graph interpretation. This in turn shows that the information flow between input and output is not optimal with an initialization common today. The lightning initialization sets the weights so that complete information paths exist between input and output from the start. It turned out that pure dense networks and also more complex networks with additional layers benefit from this initialization. The networks accuracy increases faster. The lightning initialization has two parameters which behaved robustly in the tests carried out. However, especially with more complex networks, an improvement effect only occurs at lower learning rates, which shows that the initialization retains its positive effect over the epochs with learning rate reduction.