nuCarla: A nuScenes-Style Bird's-Eye View Perception Dataset for CARLA Simulation

TL;DR

nuCarla is a large-scale, simulation-based bird's-eye view perception dataset for autonomous driving in CARLA, designed to facilitate closed-loop end-to-end learning and testing with compatibility to real-world models.

Contribution

It introduces nuCarla, a nuScenes-style BEV perception dataset within CARLA, enabling seamless transfer of perception models and supporting closed-loop autonomous driving research.

Findings

Achieves state-of-the-art detection results with BEV backbones.

Supports closed-loop simulation deployment for end-to-end autonomous driving.

Provides a large-scale, balanced dataset compatible with nuScenes format.

Abstract

End-to-end (E2E) autonomous driving heavily relies on closed-loop simulation, where perception, planning, and control are jointly trained and evaluated in interactive environments. Yet, most existing datasets are collected from the real world under non-interactive conditions, primarily supporting open-loop learning while offering limited value for closed-loop testing. Due to the lack of standardized, large-scale, and thoroughly verified datasets to facilitate learning of meaningful intermediate representations, such as bird's-eye-view (BEV) features, closed-loop E2E models remain far behind even simple rule-based baselines. To address this challenge, we introduce nuCarla, a large-scale, nuScenes-style BEV perception dataset built within the CARLA simulator. nuCarla features (1) full compatibility with the nuScenes format, enabling seamless transfer of real-world perception models; (2) a dataset scale comparable to nuScenes, but with more balanced class distributions; (3) direct usability for closed-loop simulation deployment; and (4) high-performance BEV backbones that achieve state-of-the-art detection results. By providing both data and models as open benchmarks, nuCarla substantially accelerates closed-loop E2E development, paving the way toward reliable and safety-aware research in autonomous driving.