Adaptive Random Fourier Features Training Stabilized By Resampling With Applications in Image Regression

TL;DR

This paper introduces an improved adaptive random Fourier features training algorithm that stabilizes neural network training using resampling, reduces hyperparameters, and demonstrates effectiveness in function and image regression tasks.

Contribution

The paper proposes a resampling-based enhancement to ARFF that stabilizes training, removes the need for the Metropolis test, and applies it to image regression for automated frequency sampling.

Findings

Enhanced stability in training neural networks with ARFF.

Reduced hyperparameters and computational cost.

Effective in function and image regression tasks.

Abstract

This paper presents an enhanced adaptive random Fourier features (ARFF) training algorithm for shallow neural networks, building upon the work introduced in "Adaptive Random Fourier Features with Metropolis Sampling", Kammonen et al., \emph{Foundations of Data Science}, 2(3):309--332, 2020. This improved method uses a particle filter-type resampling technique to stabilize the training process and reduce the sensitivity to parameter choices. The Metropolis test can also be omitted when resampling is used, reducing the number of hyperparameters by one and reducing the computational cost per iteration compared to the ARFF method. We present comprehensive numerical experiments demonstrating the efficacy of the proposed algorithm in function regression tasks as a stand-alone method and as a pretraining step before gradient-based optimization, using the Adam optimizer. Furthermore, we apply the proposed algorithm to a simple image regression problem, illustrating its utility in sampling frequencies for the random Fourier features (RFF) layer of coordinate-based multilayer perceptrons. In this context, we use the proposed algorithm to sample the parameters of the RFF layer in an automated manner.