Occupancy Learning with Spatiotemporal Memory

TL;DR

This paper introduces ST-Occ, a novel framework for 3D occupancy learning in autonomous driving that efficiently captures spatiotemporal information using a scene-level memory and attention mechanism, improving accuracy and consistency.

Contribution

The paper proposes a scene-level spatiotemporal memory and attention mechanism to enhance 3D occupancy prediction with temporal consistency and efficiency.

Findings

Outperforms state-of-the-art methods by 3 mIoU.

Reduces temporal inconsistency by 29%.

Effectively models historical information for better perception.

Abstract

3D occupancy becomes a promising perception representation for autonomous driving to model the surrounding environment at a fine-grained scale. However, it remains challenging to efficiently aggregate 3D occupancy over time across multiple input frames due to the high processing cost and the uncertainty and dynamics of voxels. To address this issue, we propose ST-Occ, a scene-level occupancy representation learning framework that effectively learns the spatiotemporal feature with temporal consistency. ST-Occ consists of two core designs: a spatiotemporal memory that captures comprehensive historical information and stores it efficiently through a scene-level representation and a memory attention that conditions the current occupancy representation on the spatiotemporal memory with a model of uncertainty and dynamic awareness. Our method significantly enhances the spatiotemporal representation learned for 3D occupancy prediction tasks by exploiting the temporal dependency between multi-frame inputs. Experiments show that our approach outperforms the state-of-the-art methods by a margin of 3 mIoU and reduces the temporal inconsistency by 29%.