XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis

TL;DR

This paper introduces a synthetic dataset and benchmark for evaluating novel view synthesis methods in autonomous driving, focusing on cross-lane and closed-loop scenarios that are challenging for existing approaches.

Contribution

The paper presents the first synthetic dataset specifically designed for cross-lane view synthesis in autonomous driving, along with a benchmark to evaluate existing methods under realistic simulation conditions.

Findings

Current NVS methods perform inadequately in cross-lane scenarios.

The dataset enables evaluation of NVS in more realistic autonomous driving simulations.

Significant gaps exist in existing approaches' ability to handle challenging views.

Abstract

Comprehensive testing of autonomous systems through simulation is essential to ensure the safety of autonomous driving vehicles. This requires the generation of safety-critical scenarios that extend beyond the limitations of real-world data collection, as many of these scenarios are rare or rarely encountered on public roads. However, evaluating most existing novel view synthesis (NVS) methods relies on sporadic sampling of image frames from the training data, comparing the rendered images with ground-truth images. Unfortunately, this evaluation protocol falls short of meeting the actual requirements in closed-loop simulations. Specifically, the true application demands the capability to render novel views that extend beyond the original trajectory (such as cross-lane views), which are challenging to capture in the real world. To address this, this paper presents a synthetic dataset for novel driving view synthesis evaluation, which is specifically designed for autonomous driving simulations. This unique dataset includes testing images captured by deviating from the training trajectory by $1-4$ meters. It comprises six sequences that cover various times and weather conditions. Each sequence contains $450$ training images, $120$ testing images, and their corresponding camera poses and intrinsic parameters. Leveraging this novel dataset, we establish the first realistic benchmark for evaluating existing NVS approaches under front-only and multicamera settings. The experimental findings underscore the significant gap in current approaches, revealing their inadequate ability to fulfill the demanding prerequisites of cross-lane or closed-loop simulation.