Modeling 3D Pedestrian-Vehicle Interactions for Vehicle-Conditioned Pose Forecasting

TL;DR

This paper introduces a 3D pedestrian pose forecasting framework conditioned on surrounding vehicles, enhancing prediction accuracy by explicitly modeling pedestrian-vehicle interactions with a novel dataset and network architecture.

Contribution

It presents a vehicle-conditioned 3D pedestrian pose forecasting method with a new dataset enhancement and a specialized neural network architecture for improved interaction modeling.

Findings

Significant accuracy improvements in pedestrian pose forecasting.

Effective modeling of multi-agent pedestrian-vehicle interactions.

Validation of vehicle-aware 3D pose prediction benefits for autonomous driving.

Abstract

Accurately predicting pedestrian motion is crucial for safe and reliable autonomous driving in complex urban environments. In this work, we present a 3D vehicle-conditioned pedestrian pose forecasting framework that explicitly incorporates surrounding vehicle information. To support this, we enhance the Waymo-3DSkelMo dataset with aligned 3D vehicle bounding boxes, enabling realistic modeling of multi-agent pedestrian-vehicle interactions. We introduce a sampling scheme to categorize scenes by pedestrian and vehicle count, facilitating training across varying interaction complexities. Our proposed network adapts the TBIFormer architecture with a dedicated vehicle encoder and pedestrian-vehicle interaction cross-attention module to fuse pedestrian and vehicle features, allowing predictions to be conditioned on both historical pedestrian motion and surrounding vehicles. Extensive experiments demonstrate substantial improvements in forecasting accuracy and validate different approaches for modeling pedestrian-vehicle interactions, highlighting the importance of vehicle-aware 3D pose prediction for autonomous driving. Code is available at: https://github.com/GuangxunZhu/VehCondPose3D