T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders

TL;DR

This paper introduces T-SHRED, an enhanced transformer-based model with symbolic regression for system identification, offering interpretable, efficient predictions of chaotic dynamical systems from sparse sensor data.

Contribution

The paper presents T-SHRED, integrating transformers and symbolic regression into SHRED, enabling sparse identification of nonlinear dynamics with improved interpretability and efficiency.

Findings

T-SHRED accurately predicts chaotic systems across various scales.

The model achieves interpretability through symbolic regularization.

Performance varies with data availability, excelling in low-data regimes.

Abstract

SHallow REcurrent Decoders (SHRED) are effective for system identification and forecasting from sparse sensor measurements. Such models are light-weight and computationally efficient, allowing them to be trained on consumer laptops. SHRED-based models rely on Recurrent Neural Networks (RNNs) and a simple Multi-Layer Perceptron (MLP) for the temporal encoding and spatial decoding respectively. Despite the relatively simple structure of SHRED, they are able to predict chaotic dynamical systems on different physical, spatial, and temporal scales directly from a sparse set of sensor measurements. In this work, we modify SHRED by leveraging transformers (T-SHRED) embedded with symbolic regression for the temporal encoding, circumventing auto-regressive long-term forecasting for physical data. This is achieved through a new sparse identification of nonlinear dynamics (SINDy) attention mechanism into T-SHRED to impose sparsity regularization on the latent space, which also allows for immediate symbolic interpretation. Symbolic regression improves model interpretability by learning and regularizing the dynamics of the latent space during training. We analyze the performance of T-SHRED on three different dynamical systems ranging from low-data to high-data regimes.