WaymoQA: A Multi-View Visual Question Answering Dataset for Safety-Critical Reasoning in Autonomous Driving

TL;DR

This paper introduces WaymoQA, a large dataset designed to improve multimodal reasoning in safety-critical autonomous driving scenarios, emphasizing multi-view inputs and complex high-risk situations.

Contribution

The paper defines Safety-Critical Reasoning as a new task, creates the WaymoQA dataset with 35,000 annotated questions, and demonstrates that fine-tuning models on this data enhances their reasoning in high-risk driving contexts.

Findings

Existing models underperform in safety-critical scenarios.

Fine-tuning on WaymoQA improves reasoning accuracy.

Multi-view inputs are essential for safety-critical reasoning.

Abstract

Recent advancements in multimodal large language models (MLLMs) have shown strong understanding of driving scenes, drawing interest in their application to autonomous driving. However, high-level reasoning in safety-critical scenarios, where avoiding one traffic risk can create another, remains a major challenge. Such reasoning is often infeasible with only a single front view and requires a comprehensive view of the environment, which we achieve through multi-view inputs. We define Safety-Critical Reasoning as a new task that leverages multi-view inputs to address this challenge. Then, we distill Safety-Critical Reasoning into two stages: first resolve the immediate risk, then mitigate the decision-induced downstream risks. To support this, we introduce WaymoQA, a dataset of 35,000 human-annotated question-answer pairs covering complex, high-risk driving scenarios. The dataset includes multiple-choice and open-ended formats across both image and video modalities. Experiments reveal that existing MLLMs underperform in safety-critical scenarios compared to normal scenes, but fine-tuning with WaymoQA significantly improves their reasoning ability, highlighting the effectiveness of our dataset in developing safer and more reasoning-capable driving agents. Our code and data are provided in https://github.com/sjyu001/WaymoQA