STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM

TL;DR

STD-PLM introduces a novel pre-trained language model tailored for understanding complex spatial-temporal data, enabling effective forecasting and imputation, especially in few-shot and zero-shot scenarios, by explicitly modeling correlations and topology.

Contribution

The paper presents STD-PLM, a new PLM-based framework that explicitly models spatial-temporal correlations and topology for improved forecasting and imputation tasks.

Findings

Achieves competitive performance on multiple datasets.

Demonstrates strong generalization in few-shot and zero-shot settings.

Introduces efficient sandglass attention module for better performance.

Abstract

Spatial-temporal forecasting and imputation are important for real-world intelligent systems. Most existing methods are tailored for individual forecasting or imputation tasks but are not designed for both. Additionally, they are less effective for zero-shot and few-shot learning. While pre-trained language model (PLM) have exhibited strong pattern recognition and reasoning abilities across various tasks, including few-shot and zero-shot learning, their applications in spatial-temporal data understanding has been constrained by insufficient modeling of complex correlations such as the temporal correlations, spatial connectivity, non-pairwise and high-order spatial-temporal correlations within data. In this paper, we propose STD-PLM for understanding both spatial and temporal properties of \underline{S}patial-\underline{T}emporal \underline{D}ata with \underline{PLM}, which is capable of implementing both spatial-temporal forecasting and imputation tasks. STD-PLM understands spatial-temporal correlations via explicitly designed spatial and temporal tokenizers. Topology-aware node embeddings are designed for PLM to comprehend and exploit the topology structure of data in inductive manner. Furthermore, to mitigate the efficiency issues introduced by the PLM, we design a sandglass attention module (SGA) combined with a specific constrained loss function, which significantly improves the model's efficiency while ensuring performance. Extensive experiments demonstrate that STD-PLM exhibits competitive performance and generalization capabilities across the forecasting and imputation tasks on various datasets. Moreover, STD-PLM achieves promising results on both few-shot and zero-shot tasks. The code is made available at \href{https://github.com/Hyheng/STD-PLM}{https://github.com/Hyheng/STD-PLM}