TR-LLM: Integrating Trajectory Data for Scene-Aware LLM-Based Human Action Prediction

TL;DR

This paper introduces TR-LLM, a multimodal framework that combines large language models with trajectory data to improve human action prediction in complex, occluded environments, addressing limitations of traditional video-based methods.

Contribution

The paper proposes a novel integration of trajectory data with LLMs to incorporate physical constraints, enhancing prediction accuracy in spatially complex scenarios.

Findings

Combining LLMs with trajectory data improves prediction accuracy.

The approach is especially effective with limited scene information.

Physical constraints from trajectories enhance LLM understanding.

Abstract

Accurate prediction of human behavior is crucial for AI systems to effectively support real-world applications, such as autonomous robots anticipating and assisting with human tasks. Real-world scenarios frequently present challenges such as occlusions and incomplete scene observations, which can compromise predictive accuracy. Thus, traditional video-based methods often struggle due to limited temporal and spatial perspectives. Large Language Models (LLMs) offer a promising alternative. Having been trained on a large text corpus describing human behaviors, LLMs likely encode plausible sequences of human actions in a home environment. However, LLMs, trained primarily on text data, lack inherent spatial awareness and real-time environmental perception. They struggle with understanding physical constraints and spatial geometry. Therefore, to be effective in a real-world spatial scenario, we propose a multimodal prediction framework that enhances LLM-based action prediction by integrating physical constraints derived from human trajectories. Our experiments demonstrate that combining LLM predictions with trajectory data significantly improves overall prediction performance. This enhancement is particularly notable in situations where the LLM receives limited scene information, highlighting the complementary nature of linguistic knowledge and physical constraints in understanding and anticipating human behavior.