ImputeFormer: Low Rankness-Induced Transformers for Generalizable Spatiotemporal Imputation

TL;DR

ImputeFormer introduces a low rankness-induced Transformer that combines low-rank models and deep learning to improve the accuracy, efficiency, and versatility of spatiotemporal data imputation across diverse datasets.

Contribution

The paper proposes a novel Transformer model leveraging low-rankness to balance inductive bias and expressivity for better spatiotemporal imputation.

Findings

Outperforms existing methods in accuracy and efficiency

Effective across heterogeneous datasets like traffic, solar energy, and air quality

Demonstrates the importance of low-rankness in time series modeling

Abstract

Missing data is a pervasive issue in both scientific and engineering tasks, especially for the modeling of spatiotemporal data. This problem attracts many studies to contribute to data-driven solutions. Existing imputation solutions mainly include low-rank models and deep learning models. The former assumes general structural priors but has limited model capacity. The latter possesses salient features of expressivity but lacks prior knowledge of the underlying spatiotemporal structures. Leveraging the strengths of both two paradigms, we demonstrate a low rankness-induced Transformer to achieve a balance between strong inductive bias and high model expressivity. The exploitation of the inherent structures of spatiotemporal data enables our model to learn balanced signal-noise representations, making it generalizable for a variety of imputation problems. We demonstrate its superiority in terms of accuracy, efficiency, and versatility in heterogeneous datasets, including traffic flow, solar energy, smart meters, and air quality. Promising empirical results provide strong conviction that incorporating time series primitives, such as low-rankness, can substantially facilitate the development of a generalizable model to approach a wide range of spatiotemporal imputation problems.