GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction

TL;DR

GaussianFormer introduces a sparse, object-centric 3D scene representation using semantic Gaussians, enabling efficient and accurate 3D occupancy prediction for autonomous driving with reduced memory usage.

Contribution

The paper proposes a novel sparse Gaussian-based scene representation and an efficient Gaussian-to-voxel splatting method for 3D semantic occupancy prediction.

Findings

Achieves comparable accuracy to state-of-the-art methods.

Uses significantly less memory (17.8%-24.8%).

Demonstrates effectiveness on nuScenes and KITTI-360 datasets.

Abstract

3D semantic occupancy prediction aims to obtain 3D fine-grained geometry and semantics of the surrounding scene and is an important task for the robustness of vision-centric autonomous driving. Most existing methods employ dense grids such as voxels as scene representations, which ignore the sparsity of occupancy and the diversity of object scales and thus lead to unbalanced allocation of resources. To address this, we propose an object-centric representation to describe 3D scenes with sparse 3D semantic Gaussians where each Gaussian represents a flexible region of interest and its semantic features. We aggregate information from images through the attention mechanism and iteratively refine the properties of 3D Gaussians including position, covariance, and semantics. We then propose an efficient Gaussian-to-voxel splatting method to generate 3D occupancy predictions, which only aggregates the neighboring Gaussians for a certain position. We conduct extensive experiments on the widely adopted nuScenes and KITTI-360 datasets. Experimental results demonstrate that GaussianFormer achieves comparable performance with state-of-the-art methods with only 17.8% - 24.8% of their memory consumption. Code is available at: https://github.com/huang-yh/GaussianFormer.