MGNiceNet: Unified Monocular Geometric Scene Understanding

TL;DR

MGNiceNet is a unified real-time monocular scene understanding model that combines panoptic segmentation and self-supervised depth estimation, improving accuracy and efficiency for autonomous driving applications.

Contribution

It introduces a linked kernel formulation and a panoptic-guided motion masking method to jointly perform segmentation and depth estimation without video annotations.

Findings

Achieves state-of-the-art real-time performance on Cityscapes and KITTI datasets.

Closes the gap between real-time and more computationally intensive methods.

Demonstrates effective joint segmentation and depth estimation in autonomous driving scenarios.

Abstract

Monocular geometric scene understanding combines panoptic segmentation and self-supervised depth estimation, focusing on real-time application in autonomous vehicles. We introduce MGNiceNet, a unified approach that uses a linked kernel formulation for panoptic segmentation and self-supervised depth estimation. MGNiceNet is based on the state-of-the-art real-time panoptic segmentation method RT-K-Net and extends the architecture to cover both panoptic segmentation and self-supervised monocular depth estimation. To this end, we introduce a tightly coupled self-supervised depth estimation predictor that explicitly uses information from the panoptic path for depth prediction. Furthermore, we introduce a panoptic-guided motion masking method to improve depth estimation without relying on video panoptic segmentation annotations. We evaluate our method on two popular autonomous driving datasets, Cityscapes and KITTI. Our model shows state-of-the-art results compared to other real-time methods and closes the gap to computationally more demanding methods. Source code and trained models are available at https://github.com/markusschoen/MGNiceNet.