SyntheOcc: Synthesize Geometric-Controlled Street View Images through 3D Semantic MPIs

TL;DR

SyntheOcc is a diffusion-based method that synthesizes photorealistic, geometry-controlled street view images from 3D occupancy labels, enabling unlimited, annotated datasets for autonomous driving perception tasks.

Contribution

It introduces a novel approach using 3D semantic multi-plane images to condition a diffusion model for generating controllable, multi-view street scene images from 3D occupancy data.

Findings

Generates high-quality, geometry-aligned street view images.

Produces diverse datasets for perception model training.

Effective as data augmentation on nuScenes dataset.

Abstract

The advancement of autonomous driving is increasingly reliant on high-quality annotated datasets, especially in the task of 3D occupancy prediction, where the occupancy labels require dense 3D annotation with significant human effort. In this paper, we propose SyntheOcc, which denotes a diffusion model that Synthesize photorealistic and geometric-controlled images by conditioning Occupancy labels in driving scenarios. This yields an unlimited amount of diverse, annotated, and controllable datasets for applications like training perception models and simulation. SyntheOcc addresses the critical challenge of how to efficiently encode 3D geometric information as conditional input to a 2D diffusion model. Our approach innovatively incorporates 3D semantic multi-plane images (MPIs) to provide comprehensive and spatially aligned 3D scene descriptions for conditioning. As a result, SyntheOcc can generate photorealistic multi-view images and videos that faithfully align with the given geometric labels (semantics in 3D voxel space). Extensive qualitative and quantitative evaluations of SyntheOcc on the nuScenes dataset prove its effectiveness in generating controllable occupancy datasets that serve as an effective data augmentation to perception models.