Recognition and 3D Localization of Pedestrian Actions from Monocular Video

TL;DR

This paper presents a novel framework for recognizing pedestrian actions and estimating their 3D location from monocular video, enhancing prediction of pedestrian behavior in urban traffic scenarios.

Contribution

It introduces a two-stream temporal relation network leveraging pose and RGB data for improved action recognition and a new loss-based network for 3D localization from monocular views.

Findings

Outperforms single-stream methods in action recognition on JAAD dataset.

Reduces average localization error on KITTI dataset.

Demonstrates effective qualitative results on H3D driving dataset.

Abstract

Understanding and predicting pedestrian behavior is an important and challenging area of research for realizing safe and effective navigation strategies in automated and advanced driver assistance technologies in urban scenes. This paper focuses on monocular pedestrian action recognition and 3D localization from an egocentric view for the purpose of predicting intention and forecasting future trajectory. A challenge in addressing this problem in urban traffic scenes is attributed to the unpredictable behavior of pedestrians, whereby actions and intentions are constantly in flux and depend on the pedestrians pose, their 3D spatial relations, and their interaction with other agents as well as with the environment. To partially address these challenges, we consider the importance of pose toward recognition and 3D localization of pedestrian actions. In particular, we propose an action recognition framework using a two-stream temporal relation network with inputs corresponding to the raw RGB image sequence of the tracked pedestrian as well as the pedestrian pose. The proposed method outperforms methods using a single-stream temporal relation network based on evaluations using the JAAD public dataset. The estimated pose and associated body key-points are also used as input to a network that estimates the 3D location of the pedestrian using a unique loss function. The evaluation of our 3D localization method on the KITTI dataset indicates the improvement of the average localization error as compared to existing state-of-the-art methods. Finally, we conduct qualitative tests of action recognition and 3D localization on HRI's H3D driving dataset.