OccFlowNet: Towards Self-supervised Occupancy Estimation via Differentiable Rendering and Occupancy Flow

TL;DR

This paper introduces OccFlowNet, a self-supervised method for 3D occupancy estimation using differentiable rendering and occupancy flow, reducing reliance on costly 3D labels while achieving state-of-the-art results.

Contribution

It proposes a novel self-supervised approach leveraging 2D labels, differentiable volumetric rendering, and occupancy flow to improve 3D occupancy estimation accuracy.

Findings

Achieves state-of-the-art performance with only 2D supervision.

Outperforms methods using 3D labels when combined with additional techniques.

Significantly surpasses previous occupancy estimation models.

Abstract

Semantic occupancy has recently gained significant traction as a prominent 3D scene representation. However, most existing methods rely on large and costly datasets with fine-grained 3D voxel labels for training, which limits their practicality and scalability, increasing the need for self-monitored learning in this domain. In this work, we present a novel approach to occupancy estimation inspired by neural radiance field (NeRF) using only 2D labels, which are considerably easier to acquire. In particular, we employ differentiable volumetric rendering to predict depth and semantic maps and train a 3D network based on 2D supervision only. To enhance geometric accuracy and increase the supervisory signal, we introduce temporal rendering of adjacent time steps. Additionally, we introduce occupancy flow as a mechanism to handle dynamic objects in the scene and ensure their temporal consistency. Through extensive experimentation we demonstrate that 2D supervision only is sufficient to achieve state-of-the-art performance compared to methods using 3D labels, while outperforming concurrent 2D approaches. When combining 2D supervision with 3D labels, temporal rendering and occupancy flow we outperform all previous occupancy estimation models significantly. We conclude that the proposed rendering supervision and occupancy flow advances occupancy estimation and further bridges the gap towards self-supervised learning in this domain.