WOMD-Reasoning: A Large-Scale Dataset for Interaction Reasoning in Driving

TL;DR

WOMD-Reasoning is a large-scale, multi-modal Q&A dataset focused on reasoning about traffic rule-induced interactions in driving scenarios, enabling improved analysis of driving behaviors and interactions.

Contribution

The paper introduces WOMD-Reasoning, the largest multi-modal Q&A dataset for traffic interaction reasoning, and demonstrates its application through the Motion-LLaVA model.

Findings

WOMD-Reasoning contains 3 million Q&As covering diverse driving topics.

Motion-LLaVA effectively utilizes WOMD-Reasoning for interaction reasoning.

The dataset supports applications like interaction prediction and traffic rule compliance planning.

Abstract

Language models uncover unprecedented abilities in analyzing driving scenarios, owing to their limitless knowledge accumulated from text-based pre-training. Naturally, they should particularly excel in analyzing rule-based interactions, such as those triggered by traffic laws, which are well documented in texts. However, such interaction analysis remains underexplored due to the lack of dedicated language datasets that address it. Therefore, we propose Waymo Open Motion Dataset-Reasoning (WOMD-Reasoning), a comprehensive large-scale Q&As dataset built on WOMD focusing on describing and reasoning traffic rule-induced interactions in driving scenarios. WOMD-Reasoning also presents by far the largest multi-modal Q&A dataset, with 3 million Q&As on real-world driving scenarios, covering a wide range of driving topics from map descriptions and motion status descriptions to narratives and analyses of agents' interactions, behaviors, and intentions. To showcase the applications of WOMD-Reasoning, we design Motion-LLaVA, a motion-language model fine-tuned on WOMD-Reasoning. Quantitative and qualitative evaluations are performed on WOMD-Reasoning dataset as well as the outputs of Motion-LLaVA, supporting the data quality and wide applications of WOMD-Reasoning, in interaction predictions, traffic rule compliance plannings, etc. The dataset and its vision modal extension are available on https://waymo.com/open/download/. The codes & prompts to build it are available on https://github.com/yhli123/WOMD-Reasoning.