Frozen LLMs as Map-Aware Spatio-Temporal Reasoners for Vehicle Trajectory Prediction

TL;DR

This paper presents a framework that uses frozen large language models as reasoning engines for vehicle trajectory prediction, integrating scene features and map semantics to evaluate their understanding of traffic dynamics.

Contribution

It introduces a novel approach to leverage frozen LLMs with minimal adaptation for understanding traffic scenes and predicting vehicle trajectories in autonomous driving.

Findings

LLMs can effectively incorporate map semantics for trajectory prediction.

The framework demonstrates strong generalizability across different LLM architectures.

Quantitative analysis shows the impact of multi-modal information on prediction accuracy.

Abstract

Large language models (LLMs) have recently demonstrated strong reasoning capabilities and attracted increasing research attention in the field of autonomous driving (AD). However, safe application of LLMs on AD perception and prediction still requires a thorough understanding of both the dynamic traffic agents and the static road infrastructure. To this end, this study introduces a framework to evaluate the capability of LLMs in understanding the behaviors of dynamic traffic agents and the topology of road networks. The framework leverages frozen LLMs as the reasoning engine, employing a traffic encoder to extract spatial-level scene features from observed trajectories of agents, while a lightweight Convolutional Neural Network (CNN) encodes the local high-definition (HD) maps. To assess the intrinsic reasoning ability of LLMs, the extracted scene features are then transformed into LLM-compatible tokens via a reprogramming adapter. By residing the prediction burden with the LLMs, a simpler linear decoder is applied to output future trajectories. The framework enables a quantitative analysis of the influence of multi-modal information, especially the impact of map semantics on trajectory prediction accuracy, and allows seamless integration of frozen LLMs with minimal adaptation, thereby demonstrating strong generalizability across diverse LLM architectures and providing a unified platform for model evaluation.