On the choice of the non-trainable internal weights in random feature maps

TL;DR

This paper investigates how to optimally select fixed internal weights in random feature maps for improved forecasting of dynamical systems, demonstrating that a simple algorithm can enhance performance while maintaining low computational cost.

Contribution

It introduces a hit-and-run algorithm for selecting internal weights in random feature maps, improving forecasting accuracy and highlighting the importance of feature quantity.

Findings

Number of good features correlates with forecasting skill.

Random feature maps outperform trained neural networks in forecasting.

Random feature maps require significantly less computation.

Abstract

The computationally cheap machine learning architecture of random feature maps can be viewed as a single-layer feedforward network in which the weights of the hidden layer are random but fixed and only the outer weights are learned via linear regression. The internal weights are typically chosen from a prescribed distribution. The choice of the internal weights significantly impacts the accuracy of random feature maps. We address here the task of how to best select the internal weights. In particular, we consider the forecasting problem whereby random feature maps are used to learn a one-step propagator map for a dynamical system. We provide a computationally cheap hit-and-run algorithm to select good internal weights which lead to good forecasting skill. We show that the number of good features is the main factor controlling the forecasting skill of random feature maps and acts as an effective feature dimension. Lastly, we compare random feature maps with single-layer feedforward neural networks in which the internal weights are now learned using gradient descent. We find that random feature maps have superior forecasting capabilities whilst having several orders of magnitude lower computational cost.