Rank2Tell: A Multimodal Driving Dataset for Joint Importance Ranking and Reasoning

TL;DR

This paper presents Rank2Tell, a comprehensive multimodal dataset for autonomous vehicle scene understanding, and introduces a joint model for importance ranking and explanation generation, advancing interpretability in autonomous driving systems.

Contribution

The paper introduces Rank2Tell, a novel multi-modal dataset with dense annotations for importance ranking and reasoning, and proposes a joint model for importance ranking and caption generation.

Findings

The dataset enables detailed semantic and relational scene understanding.

The joint model achieves promising results in importance ranking and captioning tasks.

Rank2Tell facilitates research on interpretability and trustworthiness in autonomous driving.

Abstract

The widespread adoption of commercial autonomous vehicles (AVs) and advanced driver assistance systems (ADAS) may largely depend on their acceptance by society, for which their perceived trustworthiness and interpretability to riders are crucial. In general, this task is challenging because modern autonomous systems software relies heavily on black-box artificial intelligence models. Towards this goal, this paper introduces a novel dataset, Rank2Tell, a multi-modal ego-centric dataset for Ranking the importance level and Telling the reason for the importance. Using various close and open-ended visual question answering, the dataset provides dense annotations of various semantic, spatial, temporal, and relational attributes of various important objects in complex traffic scenarios. The dense annotations and unique attributes of the dataset make it a valuable resource for researchers working on visual scene understanding and related fields. Furthermore, we introduce a joint model for joint importance level ranking and natural language captions generation to benchmark our dataset and demonstrate performance with quantitative evaluations.