Monocular Semantic Scene Completion via Masked Recurrent Networks

TL;DR

This paper introduces MonoMRN, a two-stage monocular semantic scene completion framework using masked recurrent networks, which improves accuracy and robustness in complex indoor and outdoor scenes.

Contribution

It proposes a novel two-stage framework with Masked Recurrent Networks, including MS-GRU and distance attention projection, to enhance scene completion accuracy and efficiency.

Findings

Achieves state-of-the-art performance on NYUv2 and SemanticKITTI datasets.

Demonstrates robustness to various disturbances in scene completion.

Supports both indoor and outdoor scene understanding.

Abstract

Monocular Semantic Scene Completion (MSSC) aims to predict the voxel-wise occupancy and semantic category from a single-view RGB image. Existing methods adopt a single-stage framework that aims to simultaneously achieve visible region segmentation and occluded region hallucination, while also being affected by inaccurate depth estimation. Such methods often achieve suboptimal performance, especially in complex scenes. We propose a novel two-stage framework that decomposes MSSC into coarse MSSC followed by the Masked Recurrent Network. Specifically, we propose the Masked Sparse Gated Recurrent Unit (MS-GRU) which concentrates on the occupied regions by the proposed mask updating mechanism, and a sparse GRU design is proposed to reduce the computation cost. Additionally, we propose the distance attention projection to reduce projection errors by assigning different attention scores according to the distance to the observed surface. Experimental results demonstrate that our proposed unified framework, MonoMRN, effectively supports both indoor and outdoor scenes and achieves state-of-the-art performance on the NYUv2 and SemanticKITTI datasets. Furthermore, we conduct robustness analysis under various disturbances, highlighting the role of the Masked Recurrent Network in enhancing the model's resilience to such challenges. The source code is publicly available.