On Transferability of Driver Observation Models from Simulated to Real Environments in Autonomous Cars

TL;DR

This study evaluates the transferability of driver observation models trained in simulation to real-world autonomous driving scenarios, highlighting significant accuracy drops and the need for more robust models.

Contribution

It introduces a real-world dataset aligned with a simulator dataset and analyzes the transferability challenges of I3D-based driver observation models.

Findings

Simulator-trained models achieve 85.7% accuracy in simulation.

Accuracy drops to 46.6% when applied to real-world data.

Transferability varies significantly across different behavior classes.

Abstract

For driver observation frameworks, clean datasets collected in controlled simulated environments often serve as the initial training ground. Yet, when deployed under real driving conditions, such simulator-trained models quickly face the problem of distributional shifts brought about by changing illumination, car model, variations in subject appearances, sensor discrepancies, and other environmental alterations. This paper investigates the viability of transferring video-based driver observation models from simulation to real-world scenarios in autonomous vehicles, given the frequent use of simulation data in this domain due to safety issues. To achieve this, we record a dataset featuring actual autonomous driving conditions and involving seven participants engaged in highly distracting secondary activities. To enable direct SIM to REAL transfer, our dataset was designed in accordance with an existing large-scale simulator dataset used as the training source. We utilize the Inflated 3D ConvNet (I3D) model, a popular choice for driver observation, with Gradient-weighted Class Activation Mapping (Grad-CAM) for detailed analysis of model decision-making. Though the simulator-based model clearly surpasses the random baseline, its recognition quality diminishes, with average accuracy dropping from 85.7% to 46.6%. We also observe strong variations across different behavior classes. This underscores the challenges of model transferability, facilitating our research of more robust driver observation systems capable of dealing with real driving conditions.