GaussianBeV: 3D Gaussian Representation meets Perception Models for BeV Segmentation

TL;DR

GaussianBeV introduces a novel 3D Gaussian-based scene representation for Bird's-eye View segmentation, significantly improving detail and accuracy by leveraging Gaussian splatting in an online, integrated manner.

Contribution

It is the first to apply 3D Gaussian modeling and rendering online for BeV perception, enhancing scene detail without scene-specific optimization.

Findings

Achieves state-of-the-art performance on nuScenes BeV segmentation

Effectively models fine scene structures with 3D Gaussians

Integrates Gaussian splatting into a single-stage perception model

Abstract

The Bird's-eye View (BeV) representation is widely used for 3D perception from multi-view camera images. It allows to merge features from different cameras into a common space, providing a unified representation of the 3D scene. The key component is the view transformer, which transforms image views into the BeV. However, actual view transformer methods based on geometry or cross-attention do not provide a sufficiently detailed representation of the scene, as they use a sub-sampling of the 3D space that is non-optimal for modeling the fine structures of the environment. In this paper, we propose GaussianBeV, a novel method for transforming image features to BeV by finely representing the scene using a set of 3D gaussians located and oriented in 3D space. This representation is then splattered to produce the BeV feature map by adapting recent advances in 3D representation rendering based on gaussian splatting. GaussianBeV is the first approach to use this 3D gaussian modeling and 3D scene rendering process online, i.e. without optimizing it on a specific scene and directly integrated into a single stage model for BeV scene understanding. Experiments show that the proposed representation is highly effective and place GaussianBeV as the new state-of-the-art on the BeV semantic segmentation task on the nuScenes dataset.