Seeing Beyond Frames: Zero-Shot Pedestrian Intention Prediction with Raw Temporal Video and Multimodal Cues

TL;DR

This paper presents BF-PIP, a zero-shot pedestrian intention prediction method that leverages continuous video and multimodal cues, achieving high accuracy without retraining, advancing autonomous driving perception capabilities.

Contribution

Introduces BF-PIP, a zero-shot approach using temporal video and multimodal prompts for pedestrian intention prediction, eliminating the need for retraining in new scenarios.

Findings

Achieves 73% prediction accuracy without additional training.

Outperforms GPT-4V baseline by 18%.

Enhances perception by combining temporal video and contextual cues.

Abstract

Pedestrian intention prediction is essential for autonomous driving in complex urban environments. Conventional approaches depend on supervised learning over frame sequences and require extensive retraining to adapt to new scenarios. Here, we introduce BF-PIP (Beyond Frames Pedestrian Intention Prediction), a zero-shot approach built upon Gemini 2.5 Pro. It infers crossing intentions directly from short, continuous video clips enriched with structured JAAD metadata. In contrast to GPT-4V based methods that operate on discrete frames, BF-PIP processes uninterrupted temporal clips. It also incorporates bounding-box annotations and ego-vehicle speed via specialized multimodal prompts. Without any additional training, BF-PIP achieves 73% prediction accuracy, outperforming a GPT-4V baseline by 18 %. These findings illustrate that combining temporal video inputs with contextual cues enhances spatiotemporal perception and improves intent inference under ambiguous conditions. This approach paves the way for agile, retraining-free perception module in intelligent transportation system.