Aspects of importance sampling in parameter selection for neural networks using ridgelet transform

TL;DR

This paper explores the use of ridgelet transform-derived oracle distributions for parameter initialization in neural networks, enabling linear regression-based construction and emphasizing the importance of weight magnitudes over intercepts.

Contribution

It introduces a novel perspective on importance sampling using ridgelet transforms and proposes new parameter sampling algorithms for neural networks.

Findings

Weight parameters are more influential than intercepts.

Oracle distributions can replace backpropagation in simple cases.

Sampling algorithms improve parameter initialization.

Abstract

The choice of parameters in neural networks is crucial in the performance, and an oracle distribution derived from the ridgelet transform enables us to obtain suitable initial parameters. In other words, the distribution of parameters is connected to the integral representation of target functions. The oracle distribution allows us to avoid the conventional backpropagation learning process; only a linear regression is enough to construct the neural network in simple cases. This study provides a new look at the oracle distributions and ridgelet transforms, i.e., an aspect of importance sampling. In addition, we propose extensions of the parameter sampling methods. We demonstrate the aspect of importance sampling and the proposed sampling algorithms via one-dimensional and high-dimensional examples; the results imply that the magnitude of weight parameters could be more crucial than the intercept parameters.