MonoRec: Semi-Supervised Dense Reconstruction in Dynamic Environments from a Single Moving Camera

TL;DR

MonoRec is a semi-supervised monocular dense reconstruction method that predicts depth maps from a single moving camera, effectively handling dynamic scenes by detecting moving objects and generalizing across datasets.

Contribution

The paper introduces MonoRec, a novel semi-supervised multi-view stereo architecture with a mask module for dynamic object reconstruction without LiDAR data.

Findings

Achieves state-of-the-art performance on KITTI dataset

Generalizes well to Oxford RobotCar and TUM-Mono datasets

Effectively reconstructs static and moving objects

Abstract

In this paper, we propose MonoRec, a semi-supervised monocular dense reconstruction architecture that predicts depth maps from a single moving camera in dynamic environments. MonoRec is based on a multi-view stereo setting which encodes the information of multiple consecutive images in a cost volume. To deal with dynamic objects in the scene, we introduce a MaskModule that predicts moving object masks by leveraging the photometric inconsistencies encoded in the cost volumes. Unlike other multi-view stereo methods, MonoRec is able to reconstruct both static and moving objects by leveraging the predicted masks. Furthermore, we present a novel multi-stage training scheme with a semi-supervised loss formulation that does not require LiDAR depth values. We carefully evaluate MonoRec on the KITTI dataset and show that it achieves state-of-the-art performance compared to both multi-view and single-view methods. With the model trained on KITTI, we further demonstrate that MonoRec is able to generalize well to both the Oxford RobotCar dataset and the more challenging TUM-Mono dataset recorded by a handheld camera. Code and related materials will be available at https://vision.in.tum.de/research/monorec.