Constructing Extreme Learning Machines with zero Spectral Bias

TL;DR

This paper investigates the spectral bias in Extreme Learning Machines (ELMs), finds that they are not inherently free of it, but demonstrates that Fourier Feature Embeddings can effectively eliminate spectral bias in ELMs, enhancing their applicability.

Contribution

The study reveals that ELMs are not naturally free of spectral bias and introduces Fourier Feature Embeddings as a method to eliminate this bias in ELMs.

Findings

ELMs are not inherently free of spectral bias.

Fourier Feature Embeddings can completely eliminate spectral bias in ELMs.

Eliminating spectral bias enables ELMs to be used effectively in high-frequency resolution tasks like PINNs.

Abstract

The phenomena of Spectral Bias, where the higher frequency components of a function being learnt in a feedforward Artificial Neural Network (ANN) are seen to converge more slowly than the lower frequencies, is observed ubiquitously across ANNs. This has created technology challenges in fields where resolution of higher frequencies is crucial, like in Physics Informed Neural Networks (PINNs). Extreme Learning Machines (ELMs) that obviate an iterative solution process which provides the theoretical basis of Spectral Bias (SB), should in principle be free of the same. This work verifies the reliability of this assumption, and shows that it is incorrect. However, the structure of ELMs makes them naturally amenable to implementation of variants of Fourier Feature Embeddings, which have been shown to mitigate SB in ANNs. This approach is implemented and verified to completely eliminate SB, thus bringing into feasibility the application of ELMs for practical problems like PINNs where resolution of higher frequencies is essential.