Can Transformers overcome the lack of data in the simulation of history-dependent flows?

TL;DR

This paper investigates the ability of Transformer neural networks to handle the lack of historical data in simulating history-dependent flows, comparing their performance to other models across various fluid dynamics benchmarks.

Contribution

It demonstrates that Transformers outperform structure-preserving neural networks in systems with missing data, especially in low-dimensional latent spaces, highlighting their potential in data-scarce scenarios.

Findings

Transformers outperform structure-preserving models with missing data.

Transformers achieve lower errors in low-dimensional latent spaces.

Structure-preserving models perform better when all variables are known.

Abstract

It is well known that the lack of information about certain variables necessary for the description of a dynamical system leads to the introduction of historical dependence (lack of Markovian character of the model) and noise. Traditionally, scientists have made up for these shortcomings by designing phenomenological variables that take into account this historical dependence (typically, conformational tensors in fluids). Often, these phenomenological variables are not easily measurable experimentally. In this work, we study to what extent Transformer architectures are able to cope with the lack of experimental data on these variables. The methodology is evaluated on three benchmark problems: a cylinder flow with no history dependence, a viscoelastic Couette flow modeled via the Oldroyd-B formalism, and a non-linear polymeric fluid described by the FENE model. Our results show that the Transformer outperforms a thermodynamically consistent, structure-preserving neural network with metriplectic bias in systems with missing experimental data, providing lower errors even in low-dimensional latent spaces. In contrast, for systems whose state variables can be fully known, the metriplectic model achieves superior performance.