LongTail Driving Scenarios with Reasoning Traces: The KITScenes LongTail Dataset

TL;DR

This paper introduces a new multimodal dataset for long-tail driving scenarios, emphasizing reasoning traces and multilingual instructions to improve generalization in self-driving models.

Contribution

The dataset provides multi-view videos, trajectories, instructions, and reasoning traces in multiple languages, enabling research on reasoning and generalization in autonomous driving.

Findings

Benchmark results show improved instruction following in multimodal models.

Multilingual reasoning traces enhance understanding across diverse cultural contexts.

Dataset facilitates studying the impact of reasoning on driving performance.

Abstract

In real-world domains such as self-driving, generalization to rare scenarios remains a fundamental challenge. To address this, we introduce a new dataset designed for end-to-end driving that focuses on long-tail driving events. We provide multi-view video data, trajectories, high-level instructions, and detailed reasoning traces, facilitating in-context learning and few-shot generalization. The resulting benchmark for multimodal models, such as VLMs and VLAs, goes beyond safety and comfort metrics by evaluating instruction following and semantic coherence between model outputs. The multilingual reasoning traces in English, Spanish, and Chinese are from domain experts with diverse cultural backgrounds. Thus, our dataset is a unique resource for studying how different forms of reasoning affect driving competence. Our dataset is available at: https://hf.co/datasets/kit-mrt/kitscenes-longtail