Learning and extrapolating scale-invariant processes

TL;DR

This paper investigates how machine learning models can predict rare, large-scale events in scale-invariant processes like earthquakes, focusing on leveraging known symmetries such as scale invariance for extrapolation.

Contribution

The study evaluates various neural network architectures, including novel wavelet and Fourier-based models, for their ability to extrapolate scale-free processes and addresses challenges related to spectral biases.

Findings

Scale-invariant models like Riesz networks perform better in extrapolation.

Spectral biases hinder the ability to predict rare large events.

Incorporating inductive biases improves model extrapolation capabilities.

Abstract

Machine Learning (ML) has deeply changed some fields recently, like Language and Vision and we may expect it to be relevant also to the analysis of of complex systems. Here we want to tackle the question of how and to which extent can one regress scale-free processes, i.e. processes displaying power law behavior, like earthquakes or avalanches? We are interested in predicting the large ones, i.e. rare events in the training set which therefore require extrapolation capabilities of the model. For this we consider two paradigmatic problems that are statistically self-similar. The first one is a 2-dimensional fractional Gaussian field obeying linear dynamics, self-similar by construction and amenable to exact analysis. The second one is the Abelian sandpile model, exhibiting self-organized criticality. The emerging paradigm of Geometric Deep Learning shows that including known symmetries into the model's architecture is key to success. Here one may hope to extrapolate only by leveraging scale invariance. This is however a peculiar symmetry, as it involves possibly non-trivial coarse-graining operations and anomalous scaling. We perform experiments on various existing architectures like U-net, Riesz network (scale invariant by construction), or our own proposals: a wavelet-decomposition based Graph Neural Network (with discrete scale symmetry), a Fourier embedding layer and a Fourier-Mellin Neural Operator. Based on these experiments and a complete characterization of the linear case, we identify the main issues relative to spectral biases and coarse-grained representations, and discuss how to alleviate them with the relevant inductive biases.