GATGPT: A Pre-trained Large Language Model with Graph Attention Network for Spatiotemporal Imputation

TL;DR

GATGPT introduces a novel framework combining graph attention mechanisms with pre-trained large language models to improve spatiotemporal data imputation, leveraging existing knowledge and fine-tuning for diverse applications.

Contribution

The paper presents a new method that integrates graph attention with LLMs for spatiotemporal imputation, reducing training complexity and enhancing spatial-temporal understanding.

Findings

Achieves comparable results to deep learning benchmarks on real-world datasets.

Effectively leverages pre-trained LLMs for spatiotemporal tasks.

Enhances spatial relationship modeling with graph attention.

Abstract

The analysis of spatiotemporal data is increasingly utilized across diverse domains, including transportation, healthcare, and meteorology. In real-world settings, such data often contain missing elements due to issues like sensor malfunctions and data transmission errors. The objective of spatiotemporal imputation is to estimate these missing values by understanding the inherent spatial and temporal relationships in the observed multivariate time series. Traditionally, spatiotemporal imputation has relied on specific, intricate architectures designed for this purpose, which suffer from limited applicability and high computational complexity. In contrast, our approach integrates pre-trained large language models (LLMs) into spatiotemporal imputation, introducing a groundbreaking framework, GATGPT. This framework merges a graph attention mechanism with LLMs. We maintain most of the LLM parameters unchanged to leverage existing knowledge for learning temporal patterns, while fine-tuning the upper layers tailored to various applications. The graph attention component enhances the LLM's ability to understand spatial relationships. Through tests on three distinct real-world datasets, our innovative approach demonstrates comparable results to established deep learning benchmarks.