Deep Virtual-to-Real Distillation for Pedestrian Crossing Prediction

TL;DR

This paper introduces a deep virtual-to-real distillation framework that leverages synthetic data to improve pedestrian crossing prediction in real driving scenarios, addressing data scarcity and diversity issues.

Contribution

The paper proposes a novel virtual-to-real distillation method using synthetic videos to enhance pedestrian crossing prediction, with a new benchmark dataset and state-of-the-art results.

Findings

Achieved superior performance on JAAD and PIE datasets.

Constructed a large synthetic video benchmark with 745k frames.

Demonstrated the effectiveness of synthetic data in real-world prediction tasks.

Abstract

Pedestrian crossing is one of the most typical behavior which conflicts with natural driving behavior of vehicles. Consequently, pedestrian crossing prediction is one of the primary task that influences the vehicle planning for safe driving. However, current methods that rely on the practically collected data in real driving scenes cannot depict and cover all kinds of scene condition in real traffic world. To this end, we formulate a deep virtual to real distillation framework by introducing the synthetic data that can be generated conveniently, and borrow the abundant information of pedestrian movement in synthetic videos for the pedestrian crossing prediction in real data with a simple and lightweight implementation. In order to verify this framework, we construct a benchmark with 4667 virtual videos owning about 745k frames (called Virtual-PedCross-4667), and evaluate the proposed method on two challenging datasets collected in real driving situations, i.e., JAAD and PIE datasets. State-of-the-art performance of this framework is demonstrated by exhaustive experiment analysis. The dataset and code can be downloaded from the website \url{http://www.lotvs.net/code_data/}.