GaussRender: Learning 3D Occupancy with Gaussian Rendering

TL;DR

GaussRender introduces a projective consistency loss using differentiable Gaussian splatting to improve 3D occupancy predictions in autonomous driving scenes, resulting in more accurate and coherent 3D reconstructions.

Contribution

The paper presents GaussRender, a novel module that enforces projective consistency via differentiable rendering, enhancing 3D occupancy learning without additional inference costs.

Findings

Significant improvements in geometric fidelity on multiple benchmarks.

State-of-the-art results on surface-sensitive metrics like RayIoU.

Effective integration with existing 3D occupancy architectures.

Abstract

Understanding the 3D geometry and semantics of driving scenes is critical for safe autonomous driving. Recent advances in 3D occupancy prediction have improved scene representation but often suffer from visual inconsistencies, leading to floating artifacts and poor surface localization. Existing voxel-wise losses (e.g., cross-entropy) fail to enforce visible geometric coherence. In this paper, we propose GaussRender, a module that improves 3D occupancy learning by enforcing projective consistency. Our key idea is to project both predicted and ground-truth 3D occupancy into 2D camera views, where we apply supervision. Our method penalizes 3D configurations that produce inconsistent 2D projections, thereby enforcing a more coherent 3D structure. To achieve this efficiently, we leverage differentiable rendering with Gaussian splatting. GaussRender seamlessly integrates with existing architectures while maintaining efficiency and requiring no inference-time modifications. Extensive evaluations on multiple benchmarks (SurroundOcc-nuScenes, Occ3D-nuScenes, SSCBench-KITTI360) demonstrate that GaussRender significantly improves geometric fidelity across various 3D occupancy models (TPVFormer, SurroundOcc, Symphonies), achieving state-of-the-art results, particularly on surface-sensitive metrics such as RayIoU. The code is open-sourced at https://github.com/valeoai/GaussRender.