Let Occ Flow: Self-Supervised 3D Occupancy Flow Prediction

TL;DR

Let Occ Flow is a pioneering self-supervised method for joint 3D occupancy and flow prediction from camera inputs, advancing autonomous perception without requiring 3D annotations.

Contribution

It introduces a novel self-supervised framework utilizing attention mechanisms and differentiable rendering for 3D scene understanding from monocular images.

Findings

Achieves competitive results on nuScenes and KITTI datasets.

Eliminates the need for 3D annotations in occupancy and flow prediction.

Effectively captures dynamic object dependencies using attention-based temporal fusion.

Abstract

Accurate perception of the dynamic environment is a fundamental task for autonomous driving and robot systems. This paper introduces Let Occ Flow, the first self-supervised work for joint 3D occupancy and occupancy flow prediction using only camera inputs, eliminating the need for 3D annotations. Utilizing TPV for unified scene representation and deformable attention layers for feature aggregation, our approach incorporates a novel attention-based temporal fusion module to capture dynamic object dependencies, followed by a 3D refine module for fine-gained volumetric representation. Besides, our method extends differentiable rendering to 3D volumetric flow fields, leveraging zero-shot 2D segmentation and optical flow cues for dynamic decomposition and motion optimization. Extensive experiments on nuScenes and KITTI datasets demonstrate the competitive performance of our approach over prior state-of-the-art methods. Our project page is available at https://eliliu2233.github.io/letoccflow/