A Gravity-informed Spatiotemporal Transformer for Human Activity Intensity Prediction

TL;DR

This paper introduces Gravityformer, a physics-informed deep learning model that incorporates gravitational laws into a spatiotemporal transformer to improve human activity intensity prediction, interpretability, and generalization.

Contribution

It proposes a novel gravity-informed attention mechanism within a transformer framework, integrating physical constraints for better spatial modeling and interpretability.

Findings

Outperforms state-of-the-art benchmarks on six real-world datasets.

Provides interpretable gravity attention matrices aligned with geographical laws.

Enhances zero-shot cross-region inference accuracy.

Abstract

Human activity intensity prediction is crucial to many location-based services. Despite tremendous progress in modeling dynamics of human activity, most existing methods overlook physical constraints of spatial interaction, leading to uninterpretable spatial correlations and over-smoothing phenomenon. To address these limitations, this work proposes a physics-informed deep learning framework, namely Gravity-informed Spatiotemporal Transformer (Gravityformer) by integrating the universal law of gravitation to refine transformer attention. Specifically, it (1) estimates two spatially explicit mass parameters based on spatiotemporal embedding feature, (2) models the spatial interaction in end-to-end neural network using proposed adaptive gravity model to learn the physical constraint, and (3) utilizes the learned spatial interaction to guide and mitigate the over-smoothing phenomenon in transformer attention. Moreover, a parallel spatiotemporal graph convolution transformer is proposed for achieving a balance between coupled spatial and temporal learning. Systematic experiments on six real-world large-scale activity datasets demonstrate the quantitative and qualitative superiority of our model over state-of-the-art benchmarks. Additionally, the learned gravity attention matrix can be not only disentangled and interpreted based on geographical laws, but also improved the generalization in zero-shot cross-region inference. This work provides a novel insight into integrating physical laws with deep learning for spatiotemporal prediction.