MulCPred: Learning Multi-modal Concepts for Explainable Pedestrian Action Prediction

TL;DR

MulCPred is a novel multi-modal concept learning framework that enhances explainability in pedestrian action prediction by integrating local, diverse concepts, and demonstrating improved interpretability and generalizability across datasets.

Contribution

This paper introduces MulCPred, a multi-modal concept-based framework that provides explainable pedestrian action predictions with locality, diversity, and cross-dataset robustness.

Findings

Improves explainability of pedestrian action prediction models.

Maintains competitive prediction performance.

Enhances cross-dataset generalization by removing unrecognizable concepts.

Abstract

Pedestrian action prediction is of great significance for many applications such as autonomous driving. However, state-of-the-art methods lack explainability to make trustworthy predictions. In this paper, a novel framework called MulCPred is proposed that explains its predictions based on multi-modal concepts represented by training samples. Previous concept-based methods have limitations including: 1) they cannot directly apply to multi-modal cases; 2) they lack locality to attend to details in the inputs; 3) they suffer from mode collapse. These limitations are tackled accordingly through the following approaches: 1) a linear aggregator to integrate the activation results of the concepts into predictions, which associates concepts of different modalities and provides ante-hoc explanations of the relevance between the concepts and the predictions; 2) a channel-wise recalibration module that attends to local spatiotemporal regions, which enables the concepts with locality; 3) a feature regularization loss that encourages the concepts to learn diverse patterns. MulCPred is evaluated on multiple datasets and tasks. Both qualitative and quantitative results demonstrate that MulCPred is promising in improving the explainability of pedestrian action prediction without obvious performance degradation. Furthermore, by removing unrecognizable concepts from MulCPred, the cross-dataset prediction performance is improved, indicating the feasibility of further generalizability of MulCPred.