A Spatiotemporal Approach to Tri-Perspective Representation for 3D Semantic Occupancy Prediction

TL;DR

This paper introduces S2TPVFormer, a spatiotemporal transformer that leverages temporal cues via a novel attention mechanism to improve vision-based 3D semantic occupancy prediction, showing significant performance gains on nuScenes.

Contribution

It proposes a new spatiotemporal transformer architecture with a Temporal Cross-View Hybrid Attention mechanism for enhanced 3D scene understanding.

Findings

Achieved +4.1% mIoU improvement on nuScenes dataset.

Demonstrated the effectiveness of incorporating temporal cues in 3D occupancy prediction.

Validated the approach's superiority over baseline TPVFormer.

Abstract

Holistic understanding and reasoning in 3D scenes are crucial for the success of autonomous driving systems. The evolution of 3D semantic occupancy prediction as a pretraining task for autonomous driving and robotic applications captures finer 3D details compared to traditional 3D detection methods. Vision-based 3D semantic occupancy prediction is increasingly overlooked in favor of LiDAR-based approaches, which have shown superior performance in recent years. However, we present compelling evidence that there is still potential for enhancing vision-based methods. Existing approaches predominantly focus on spatial cues such as tri-perspective view (TPV) embeddings, often overlooking temporal cues. This study introduces S2TPVFormer, a spatiotemporal transformer architecture designed to predict temporally coherent 3D semantic occupancy. By introducing temporal cues through a novel Temporal Cross-View Hybrid Attention mechanism (TCVHA), we generate Spatiotemporal TPV (S2TPV) embeddings that enhance the prior process. Experimental evaluations on the nuScenes dataset demonstrate a significant +4.1% of absolute gain in mean Intersection over Union (mIoU) for 3D semantic occupancy compared to baseline TPVFormer, validating the effectiveness of S2TPVFormer in advancing 3D scene perception.