Predicting Take-over Time for Autonomous Driving with Real-World Data: Robust Data Augmentation, Models, and Evaluation

TL;DR

This paper presents a robust data augmentation scheme and models for predicting take-over times in autonomous driving, utilizing real-world driver data and computer vision features to improve safety and responsiveness.

Contribution

The study introduces a novel data augmentation method for driver control transition datasets and develops models that outperform baseline approaches using multi-level visual features.

Findings

Augmented dataset improves model performance

Multi-level visual features enhance prediction accuracy

Models can estimate take-over times continuously in real-world scenarios

Abstract

Understanding occupant-vehicle interactions by modeling control transitions is important to ensure safe approaches to passenger vehicle automation. Models which contain contextual, semantically meaningful representations of driver states can be used to determine the appropriate timing and conditions for transfer of control between driver and vehicle. However, such models rely on real-world control take-over data from drivers engaged in distracting activities, which is costly to collect. Here, we introduce a scheme for data augmentation for such a dataset. Using the augmented dataset, we develop and train take-over time (TOT) models that operate sequentially on mid and high-level features produced by computer vision algorithms operating on different driver-facing camera views, showing models trained on the augmented dataset to outperform the initial dataset. The demonstrated model features encode different aspects of the driver state, pertaining to the face, hands, foot and upper body of the driver. We perform ablative experiments on feature combinations as well as model architectures, showing that a TOT model supported by augmented data can be used to produce continuous estimates of take-over times without delay, suitable for complex real-world scenarios.