Pedestrian Crossing Intention Prediction Using Multimodal Fusion Network

TL;DR

This paper introduces a multimodal fusion network utilizing visual and motion data with Transformer modules and attention mechanisms to accurately predict pedestrian crossing intentions, enhancing autonomous vehicle safety.

Contribution

The paper presents a novel multimodal fusion network with depth-guided, modality, and temporal attention modules for improved pedestrian intention prediction.

Findings

Achieves superior performance on JAAD dataset

Effectively integrates multiple modalities for prediction

Outperforms baseline methods in accuracy

Abstract

Pedestrian crossing intention prediction is essential for the deployment of autonomous vehicles (AVs) in urban environments. Ideal prediction provides AVs with critical environmental cues, thereby reducing the risk of pedestrian-related collisions. However, the prediction task is challenging due to the diverse nature of pedestrian behavior and its dependence on multiple contextual factors. This paper proposes a multimodal fusion network that leverages seven modality features from both visual and motion branches, aiming to effectively extract and integrate complementary cues across different modalities. Specifically, motion and visual features are extracted from the raw inputs using multiple Transformer-based extraction modules. Depth-guided attention module leverages depth information to guide attention towards salient regions in another modality through comprehensive spatial feature interactions. To account for the varying importance of different modalities and frames, modality attention and temporal attention are designed to selectively emphasize informative modalities and effectively capture temporal dependencies. Extensive experiments on the JAAD dataset validate the effectiveness of the proposed network, achieving superior performance compared to the baseline methods.