Representation Learning for Spatiotemporal Physical Systems

TL;DR

This paper investigates how different self-supervised learning methods can create meaningful representations of spatiotemporal physical systems that are useful for downstream scientific tasks, highlighting the superiority of latent space learning methods.

Contribution

It evaluates various self-supervised learning approaches for physical systems and demonstrates that latent space methods outperform pixel-level prediction models in downstream tasks.

Findings

Latent space learning methods outperform pixel-level prediction models.

Not all physics modeling methods are equally effective for downstream tasks.

Self-supervised learning can produce physically relevant representations.

Abstract

Machine learning approaches to spatiotemporal physical systems have primarily focused on next-frame prediction, with the goal of learning an accurate emulator for the system's evolution in time. However, these emulators are computationally expensive to train and are subject to performance pitfalls, such as compounding errors during autoregressive rollout. In this work, we take a different perspective and look at scientific tasks further downstream of predicting the next frame, such as estimation of a system's governing physical parameters. Accuracy on these tasks offers a uniquely quantifiable glimpse into the physical relevance of the representations of these models. We evaluate the effectiveness of general-purpose self-supervised methods in learning physics-grounded representations that are useful for downstream scientific tasks. Surprisingly, we find that not all methods designed for physical modeling outperform generic self-supervised learning methods on these tasks, and methods that learn in the latent space (e.g., joint embedding predictive architectures, or JEPAs) outperform those optimizing pixel-level prediction objectives. Code is available at https://github.com/helenqu/physical-representation-learning.