ADAS-TO: A Large-Scale Multimodal Naturalistic Dataset and Empirical Characterization of Human Takeovers during ADAS Engagement

TL;DR

This paper introduces ADAS-TO, a comprehensive large-scale dataset capturing real-world driver takeovers during ADAS use, and provides an empirical analysis of safety-critical events and early warning cues.

Contribution

The paper presents the first extensive naturalistic dataset focused on ADAS-to-manual transitions and analyzes hazard cues and intervention dynamics in critical scenarios.

Findings

Most critical takeovers show early visual cues at least 3 seconds before intervention.

Distinct kinematic signatures are associated with different traffic and environmental conditions.

Over half of safety-critical cases exhibit actionable visual information prior to takeover.

Abstract

Takeovers remain a key safety vulnerability in production ADAS, yet existing public resources rarely provide takeover-centered, real-world data. We present ADAS-TO, the first large-scale naturalistic dataset dedicated to ADAS-to-manual transitions, containing 15,659 takeover-centered 20s clips from 327 drivers across 22 vehicle brands. Each clip synchronizes front-view video with CAN logs. Takeovers are defined as ADAS ON $\rightarrow$ OFF transitions, with the primary trigger labeled as brake, steer, gas, mixed, or system disengagement. We further separate planned driver-initiated terminations (Ego) from forced takeovers (Non-ego) using a rule-based partition. While most events occur within conservative kinematic margins, we identify a long tail of 285 safety-critical cases. For these events, we combine kinematic screening with vision--language (VLM) annotation to attribute hazards and relate them to intervention dynamics. The resulting cross-modal analysis shows distinct kinematic signatures across traffic dynamics, infrastructure degradation, and adverse environments, and finds that in 59.3% of critical cases, actionable visual cues emerge at least 3s before takeover, supporting the potential for semantics-aware early warning beyond late-stage kinematic triggers. The dataset is publicly released at huggingface.co/datasets/HenryYHW/ADAS-TO-Sample.