PIPE: Physics-Informed Position Encoding for Alignment of Satellite Images and Time Series

TL;DR

This paper introduces PIPE, a physics-informed positional encoding method that enhances the alignment and forecasting accuracy of satellite images and time series data by embedding physical information into vision language models.

Contribution

PIPE is a novel, lightweight encoding scheme that incorporates physical and sequential information into vision language models for improved multimodal time series forecasting.

Findings

Achieves state-of-the-art typhoon forecasting accuracy

Improves multimodal alignment in satellite image and time series data

Demonstrates 12% performance boost over prior methods

Abstract

Multimodal time series forecasting is foundational in various fields, such as utilizing satellite imagery and numerical data for predicting typhoons in climate science. However, existing multimodal approaches primarily focus on utilizing text data to help time series forecasting, leaving the visual data in existing time series datasets untouched. Furthermore, it is challenging for models to effectively capture the physical information embedded in visual data, such as satellite imagery's temporal and geospatial context, which extends beyond images themselves. To address this gap, we propose physics-informed positional encoding (PIPE), a lightweight method that embeds physical information into vision language models (VLMs). PIPE introduces two key innovations: (1) a physics-informed positional indexing scheme for mapping physics to positional IDs, and (2) a variant-frequency positional encoding mechanism for encoding frequency information of physical variables and sequential order of tokens within the embedding space. By preserving both the physical information and sequential order information, PIPE significantly improves multimodal alignment and forecasting accuracy. Through the experiments on the most representative and the largest open-sourced satellite image dataset, PIPE achieves state-of-the-art performance in both deep learning forecasting and climate domain methods, demonstrating superiority across benchmarks, including a 12% improvement in typhoon intensity forecasting over prior works. Our code is provided in the supplementary material.