R3DPA: Leveraging 3D Representation Alignment and RGB Pretrained Priors for LiDAR Scene Generation

TL;DR

R3DPA introduces a novel LiDAR scene generation method that leverages image-pretrained priors and self-supervised 3D features to improve realism and enable control, achieving state-of-the-art results on KITTI-360.

Contribution

It is the first to combine image-pretrained priors with self-supervised 3D features for LiDAR scene synthesis, enhancing quality and control.

Findings

Achieves state-of-the-art performance on KITTI-360.

Enables point cloud control such as inpainting and scene mixing.

Significantly improves generation quality through feature alignment.

Abstract

LiDAR scene synthesis is an emerging solution to scarcity in 3D data for robotic tasks such as autonomous driving. Recent approaches employ diffusion or flow matching models to generate realistic scenes, but 3D data remains limited compared to RGB datasets with millions of samples. We introduce R3DPA, the first LiDAR scene generation method to unlock image-pretrained priors for LiDAR point clouds, and leverage self-supervised 3D representations for state-of-the-art results. Specifically, we (i) align intermediate features of our generative model with self-supervised 3D features, which substantially improves generation quality; (ii) transfer knowledge from large-scale image-pretrained generative models to LiDAR generation, mitigating limited LiDAR datasets; and (iii) enable point cloud control at inference for object inpainting and scene mixing with solely an unconditional model. On the KITTI-360 benchmark R3DPA achieves state of the art performance. Code and pretrained models are available at https://github.com/valeoai/R3DPA.