Neural network initialization with nonlinear characteristics and information on hierarchical features

TL;DR

This paper introduces a novel neural network initialization method that leverages hierarchical feature information, improving training performance by capturing low- and high-frequency components at different layers.

Contribution

It proposes a framework adjusting SWIM algorithm parameters to incorporate hierarchical features, enhancing initialization effectiveness.

Findings

Outperforms conventional initialization algorithms on regression and classification tasks.

Highlights the importance of hierarchical features in neural network training.

Demonstrates improved learning efficiency with the proposed method.

Abstract

Initialization of neural network parameters, such as weights and biases, has a crucial impact on learning performance; if chosen well, we can even avoid the need for additional training with backpropagation. For example, algorithms based on the ridgelet transform or the SWIM (sampling where it matters) concept have been proposed for initialization. On the other hand, some works show hierarchical features in trained neural networks; neural networks tend to learn coarse information in the early-stage hidden layers. In this work, we investigate the effects of utilizing information on the hierarchical features in the initialization of neural networks. Hence, we propose a framework that adjusts the scale factors in the SWIM algorithm to capture low-frequency components in the early-stage hidden layers and to represent high-frequency components in the late-stage hidden layers. Numerical experiments on a one-dimensional regression task and the MNIST classification task demonstrate that the proposed method outperforms the conventional initialization algorithms. This work clarifies the importance of intrinsic hierarchical features in learning neural networks, and the finding yields an effective parameter initialization strategy that enhances their training performance.