Can Reasons Help Improve Pedestrian Intent Estimation? A Cross-Modal Approach

TL;DR

This paper introduces a cross-modal approach that predicts pedestrian intent and the reasons behind it, enhancing safety in autonomous navigation by making predictions more interpretable and accurate.

Contribution

It presents a novel dataset with textual explanations for pedestrian intent and a multi-task framework that jointly predicts intent and reasons, improving prediction accuracy.

Findings

5.6% and 7% accuracy and F1-score improvements on PIE++ dataset

4.4% accuracy improvement on JAAD dataset

Effective in providing human-understandable explanations for pedestrian intent

Abstract

With the increased importance of autonomous navigation systems has come an increasing need to protect the safety of Vulnerable Road Users (VRUs) such as pedestrians. Predicting pedestrian intent is one such challenging task, where prior work predicts the binary cross/no-cross intention with a fusion of visual and motion features. However, there has been no effort so far to hedge such predictions with human-understandable reasons. We address this issue by introducing a novel problem setting of exploring the intuitive reasoning behind a pedestrian's intent. In particular, we show that predicting the 'WHY' can be very useful in understanding the 'WHAT'. To this end, we propose a novel, reason-enriched PIE++ dataset consisting of multi-label textual explanations/reasons for pedestrian intent. We also introduce a novel multi-task learning framework called MINDREAD, which leverages a cross-modal representation learning framework for predicting pedestrian intent as well as the reason behind the intent. Our comprehensive experiments show significant improvement of 5.6% and 7% in accuracy and F1-score for the task of intent prediction on the PIE++ dataset using MINDREAD. We also achieved a 4.4% improvement in accuracy on a commonly used JAAD dataset. Extensive evaluation using quantitative/qualitative metrics and user studies shows the effectiveness of our approach.