Pedestrian Crossing Action Recognition and Trajectory Prediction with 3D Human Keypoints

TL;DR

This paper introduces a multi-task learning framework that leverages 3D human keypoints for pedestrian crossing action recognition and trajectory prediction, significantly improving accuracy in urban autonomous vehicle scenarios.

Contribution

The work presents a novel multi-task learning approach using 3D human keypoints, auxiliary tasks, and contrastive learning to enhance pedestrian behavior understanding.

Findings

Achieves state-of-the-art performance on multiple datasets.

Auxiliary tasks and contrastive learning improve keypoint representation.

Detailed ablation study validates each component's effectiveness.

Abstract

Accurate understanding and prediction of human behaviors are critical prerequisites for autonomous vehicles, especially in highly dynamic and interactive scenarios such as intersections in dense urban areas. In this work, we aim at identifying crossing pedestrians and predicting their future trajectories. To achieve these goals, we not only need the context information of road geometry and other traffic participants but also need fine-grained information of the human pose, motion and activity, which can be inferred from human keypoints. In this paper, we propose a novel multi-task learning framework for pedestrian crossing action recognition and trajectory prediction, which utilizes 3D human keypoints extracted from raw sensor data to capture rich information on human pose and activity. Moreover, we propose to apply two auxiliary tasks and contrastive learning to enable auxiliary supervisions to improve the learned keypoints representation, which further enhances the performance of major tasks. We validate our approach on a large-scale in-house dataset, as well as a public benchmark dataset, and show that our approach achieves state-of-the-art performance on a wide range of evaluation metrics. The effectiveness of each model component is validated in a detailed ablation study.