ProDepth: Boosting Self-Supervised Multi-Frame Monocular Depth with Probabilistic Fusion

TL;DR

ProDepth introduces a probabilistic fusion framework for self-supervised multi-frame monocular depth estimation, effectively handling dynamic scene inconsistencies by modeling uncertainty and adaptively refining depth predictions.

Contribution

It proposes a novel probabilistic approach with an auxiliary decoder and cost volume modulation to improve depth estimation in dynamic scenes, outperforming existing methods.

Findings

Outperforms state-of-the-art on Cityscapes and KITTI datasets.

Demonstrates superior generalization on Waymo Open dataset.

Effectively mitigates dynamic object interference in depth estimation.

Abstract

Self-supervised multi-frame monocular depth estimation relies on the geometric consistency between successive frames under the assumption of a static scene. However, the presence of moving objects in dynamic scenes introduces inevitable inconsistencies, causing misaligned multi-frame feature matching and misleading self-supervision during training. In this paper, we propose a novel framework called ProDepth, which effectively addresses the mismatch problem caused by dynamic objects using a probabilistic approach. We initially deduce the uncertainty associated with static scene assumption by adopting an auxiliary decoder. This decoder analyzes inconsistencies embedded in the cost volume, inferring the probability of areas being dynamic. We then directly rectify the erroneous cost volume for dynamic areas through a Probabilistic Cost Volume Modulation (PCVM) module. Specifically, we derive probability distributions of depth candidates from both single-frame and multi-frame cues, modulating the cost volume by adaptively fusing those distributions based on the inferred uncertainty. Additionally, we present a self-supervision loss reweighting strategy that not only masks out incorrect supervision with high uncertainty but also mitigates the risks in remaining possible dynamic areas in accordance with the probability. Our proposed method excels over state-of-the-art approaches in all metrics on both Cityscapes and KITTI datasets, and demonstrates superior generalization ability on the Waymo Open dataset.