RaLD: Generating High-Resolution 3D Radar Point Clouds with Latent Diffusion

TL;DR

RaLD introduces a novel latent diffusion framework that generates dense, high-resolution 3D radar point clouds from raw spectrum data, enhancing perception robustness in autonomous systems.

Contribution

It is the first to adapt latent diffusion models for radar 3D point cloud generation using scene-level autoencoding and spectrum conditioning.

Findings

RaLD produces dense, accurate 3D radar point clouds.

It outperforms existing methods in detail preservation.

The approach improves perception in adverse conditions.

Abstract

Millimeter-wave radar offers a promising sensing modality for autonomous systems thanks to its robustness in adverse conditions and low cost. However, its utility is significantly limited by the sparsity and low resolution of radar point clouds, which poses challenges for tasks requiring dense and accurate 3D perception. Despite that recent efforts have shown great potential by exploring generative approaches to address this issue, they often rely on dense voxel representations that are inefficient and struggle to preserve structural detail. To fill this gap, we make the key observation that latent diffusion models (LDMs), though successful in other modalities, have not been effectively leveraged for radar-based 3D generation due to a lack of compatible representations and conditioning strategies. We introduce RaLD, a framework that bridges this gap by integrating scene-level frustum-based LiDAR autoencoding, order-invariant latent representations, and direct radar spectrum conditioning. These insights lead to a more compact and expressive generation process. Experiments show that RaLD produces dense and accurate 3D point clouds from raw radar spectrums, offering a promising solution for robust perception in challenging environments.