Interpretable Spatial-Temporal Fusion Transformers: Multi-Output Prediction for Parametric Dynamical Systems with Time-Varying Inputs

TL;DR

This paper introduces an interpretable multi-output transformer model designed for accurate prediction of complex parametric dynamical systems with time-varying inputs, enhancing understanding of spatial-temporal interactions.

Contribution

It extends existing transformer models to handle multiple outputs and provides interpretability of spatial-temporal correlations in dynamical systems.

Findings

Accurately predicts multiple outputs in nonlinear systems.

Explores interactions between outputs via attention weights.

Handles high-dimensional parameter spaces effectively.

Abstract

We explore the promising performance of a transformer model in predicting outputs of parametric dynamical systems with external time-varying input signals. The outputs of such systems vary not only with physical parameters but also with external time-varying input signals. Accurately catching the dynamics of such systems is challenging. We have adapted and extended an existing transformer model for single output prediction to a multiple-output transformer that is able to predict multiple output responses of these systems. The multiple-output transformer generalizes the interpretability of the original transformer. The generalized interpretable attention weight matrix explores not only the temporal correlations in the sequence, but also the interactions between the multiple outputs, providing explanation for the spatial correlation in the output domain. This multiple-output transformer accurately predicts the sequence of multiple outputs, regardless of the nonlinearity of the system and the dimensionality of the parameter space.