DiffSemanticFusion: Semantic Raster BEV Fusion for Autonomous Driving via Online HD Map Diffusion

TL;DR

DiffSemanticFusion introduces a novel fusion framework combining raster and graph-based map representations with a diffusion module, significantly improving trajectory prediction and autonomous driving performance in real-world benchmarks.

Contribution

The paper proposes DiffSemanticFusion, a new multimodal fusion framework with a map diffusion module that enhances online HD map stability and expressiveness for autonomous driving.

Findings

Achieves 5.1% performance improvement on nuScenes trajectory prediction.

Attains 15% performance gain in NavHard scenarios for autonomous driving.

Map diffusion module can be integrated into other vector-based approaches to boost performance.

Abstract

Autonomous driving requires accurate scene understanding, including road geometry, traffic agents, and their semantic relationships. In online HD map generation scenarios, raster-based representations are well-suited to vision models but lack geometric precision, while graph-based representations retain structural detail but become unstable without precise maps. To harness the complementary strengths of both, we propose DiffSemanticFusion -- a fusion framework for multimodal trajectory prediction and planning. Our approach reasons over a semantic raster-fused BEV space, enhanced by a map diffusion module that improves both the stability and expressiveness of online HD map representations. We validate our framework on two downstream tasks: trajectory prediction and planning-oriented end-to-end autonomous driving. Experiments on real-world autonomous driving benchmarks, nuScenes and NAVSIM, demonstrate improved performance over several state-of-the-art methods. For the prediction task on nuScenes, we integrate DiffSemanticFusion with the online HD map informed QCNet, achieving a 5.1\% performance improvement. For end-to-end autonomous driving in NAVSIM, DiffSemanticFusion achieves state-of-the-art results, with a 15\% performance gain in NavHard scenarios. In addition, extensive ablation and sensitivity studies show that our map diffusion module can be seamlessly integrated into other vector-based approaches to enhance performance. All artifacts are available at https://github.com/SunZhigang7/DiffSemanticFusion.