SOccDPT: Semi-Supervised 3D Semantic Occupancy from Dense Prediction Transformers trained under memory constraints

TL;DR

SOccDPT is a memory-efficient semi-supervised transformer-based model that predicts 3D semantic occupancy from monocular images, trained on unstructured datasets, and outperforms existing methods in unstructured traffic scenarios.

Contribution

We introduce SOccDPT, a novel semi-supervised dense prediction transformer model trained on unstructured datasets with patch-wise training for memory efficiency.

Findings

RMSE score of 9.1473 for disparity estimation

Semantic segmentation IoU of 46.02%

Operates at 69.47 Hz

Abstract

We present SOccDPT, a memory-efficient approach for 3D semantic occupancy prediction from monocular image input using dense prediction transformers. To address the limitations of existing methods trained on structured traffic datasets, we train our model on unstructured datasets including the Indian Driving Dataset and Bengaluru Driving Dataset. Our semi-supervised training pipeline allows SOccDPT to learn from datasets with limited labels by reducing the requirement for manual labelling by substituting it with pseudo-ground truth labels to produce our Bengaluru Semantic Occupancy Dataset. This broader training enhances our model's ability to handle unstructured traffic scenarios effectively. To overcome memory limitations during training, we introduce patch-wise training where we select a subset of parameters to train each epoch, reducing memory usage during auto-grad graph construction. In the context of unstructured traffic and memory-constrained training and inference, SOccDPT outperforms existing disparity estimation approaches as shown by the RMSE score of 9.1473, achieves a semantic segmentation IoU score of 46.02% and operates at a competitive frequency of 69.47 Hz. We make our code and semantic occupancy dataset public.