JaywalkerVR: A VR System for Collecting Safety-Critical Pedestrian-Vehicle Interactions

TL;DR

JaywalkerVR is a virtual reality system designed to safely and efficiently collect rare, safety-critical pedestrian-vehicle interaction data, which enhances autonomous vehicle models' robustness.

Contribution

The paper introduces JaywalkerVR, a novel VR-based data collection system that generates high-quality, safety-critical pedestrian-vehicle interaction data for autonomous driving.

Findings

Models trained with CARLA-VR data show 10.7% lower displacement error.

Collision rate decreases by 4.9% with CARLA-VR data.

CARLA-VR improves robustness in rare scenarios.

Abstract

Developing autonomous vehicles that can safely interact with pedestrians requires large amounts of pedestrian and vehicle data in order to learn accurate pedestrian-vehicle interaction models. However, gathering data that include crucial but rare scenarios - such as pedestrians jaywalking into heavy traffic - can be costly and unsafe to collect. We propose a virtual reality human-in-the-loop simulator, JaywalkerVR, to obtain vehicle-pedestrian interaction data to address these challenges. Our system enables efficient, affordable, and safe collection of long-tail pedestrian-vehicle interaction data. Using our proposed simulator, we create a high-quality dataset with vehicle-pedestrian interaction data from safety critical scenarios called CARLA-VR. The CARLA-VR dataset addresses the lack of long-tail data samples in commonly used real world autonomous driving datasets. We demonstrate that models trained with CARLA-VR improve displacement error and collision rate by 10.7% and 4.9%, respectively, and are more robust in rare vehicle-pedestrian scenarios.