DualDiff: Dual-branch Diffusion Model for Autonomous Driving with Semantic Fusion

TL;DR

DualDiff is a novel dual-branch diffusion model that leverages semantic-rich 3D representations and a fusion mechanism to improve multi-view driving scene reconstruction and understanding.

Contribution

It introduces Occupancy Ray Sampling and Semantic Fusion Attention for better multi-modal integration in scene generation.

Findings

State-of-the-art FID score in scene generation

Improved BEV segmentation results

Enhanced 3D object detection accuracy

Abstract

Accurate and high-fidelity driving scene reconstruction relies on fully leveraging scene information as conditioning. However, existing approaches, which primarily use 3D bounding boxes and binary maps for foreground and background control, fall short in capturing the complexity of the scene and integrating multi-modal information. In this paper, we propose DualDiff, a dual-branch conditional diffusion model designed to enhance multi-view driving scene generation. We introduce Occupancy Ray Sampling (ORS), a semantic-rich 3D representation, alongside numerical driving scene representation, for comprehensive foreground and background control. To improve cross-modal information integration, we propose a Semantic Fusion Attention (SFA) mechanism that aligns and fuses features across modalities. Furthermore, we design a foreground-aware masked (FGM) loss to enhance the generation of tiny objects. DualDiff achieves state-of-the-art performance in FID score, as well as consistently better results in downstream BEV segmentation and 3D object detection tasks.