One at a Time: Progressive Multi-step Volumetric Probability Learning for Reliable 3D Scene Perception

TL;DR

This paper introduces a progressive multi-step diffusion-based framework for 3D scene perception, improving volumetric probability learning in challenging regions and surpassing LiDAR methods on SemanticKITTI.

Contribution

It proposes a novel multi-step diffusion framework, VPD, for refining 3D volumetric probabilities, addressing limitations of single-step methods in complex scene regions.

Findings

Outperforms existing methods in multi-view stereo and semantic scene completion.

First to surpass LiDAR-based methods on SemanticKITTI for SSC.

Demonstrates improved accuracy in occluded and reflective regions.

Abstract

Numerous studies have investigated the pivotal role of reliable 3D volume representation in scene perception tasks, such as multi-view stereo (MVS) and semantic scene completion (SSC). They typically construct 3D probability volumes directly with geometric correspondence, attempting to fully address the scene perception tasks in a single forward pass. However, such a single-step solution makes it hard to learn accurate and convincing volumetric probability, especially in challenging regions like unexpected occlusions and complicated light reflections. Therefore, this paper proposes to decompose the complicated 3D volume representation learning into a sequence of generative steps to facilitate fine and reliable scene perception. Considering the recent advances achieved by strong generative diffusion models, we introduce a multi-step learning framework, dubbed as VPD, dedicated to progressively refining the Volumetric Probability in a Diffusion process. Extensive experiments are conducted on scene perception tasks including multi-view stereo (MVS) and semantic scene completion (SSC), to validate the efficacy of our method in learning reliable volumetric representations. Notably, for the SSC task, our work stands out as the first to surpass LiDAR-based methods on the SemanticKITTI dataset.