Is attention to bounding boxes all you need for pedestrian action prediction?

TL;DR

This paper introduces a Transformer-based framework that predicts pedestrian crossing actions using only bounding boxes, achieving state-of-the-art accuracy on real and simulated datasets, and highlights the importance of human attention in this task.

Contribution

It demonstrates that simple bounding box inputs can outperform previous methods and shows the benefits of pre-training on simulated data for pedestrian action prediction.

Findings

Achieved 91% accuracy and 0.83 F1-score on PIE dataset.

Achieved 91% accuracy and 0.91 F1-score on CP2A dataset.

Pre-training on simulated data improves real-world prediction performance.

Abstract

The human driver is no longer the only one concerned with the complexity of the driving scenarios. Autonomous vehicles (AV) are similarly becoming involved in the process. Nowadays, the development of AVs in urban places raises essential safety concerns for vulnerable road users (VRUs) such as pedestrians. Therefore, to make the roads safer, it is critical to classify and predict the pedestrians' future behavior. In this paper, we present a framework based on multiple variations of the Transformer models able to infer predict the pedestrian street-crossing decision-making based on the dynamics of its initiated trajectory. We showed that using solely bounding boxes as input features can outperform the previous state-of-the-art results by reaching a prediction accuracy of 91\% and an F1-score of 0.83 on the PIE dataset. In addition, we introduced a large-size simulated dataset (CP2A) using CARLA for action prediction. Our model has similarly reached high accuracy (91\%) and F1-score (0.91) on this dataset. Interestingly, we showed that pre-training our Transformer model on the CP2A dataset and then fine-tuning it on the PIE dataset is beneficial for the action prediction task. Finally, our model's results are successfully supported by the "human attention to bounding boxes" experiment which we created to test humans ability for pedestrian action prediction without the need for environmental context. The code for the dataset and the models is available at: https://github.com/linaashaji/Action_Anticipation