CroMo: Cross-Modal Learning for Monocular Depth Estimation

TL;DR

This paper introduces CroMo, a novel cross-modal learning approach for monocular depth estimation that integrates polarisation, ToF, and structured-light signals, utilizing a new dataset and self-supervised training to improve accuracy and robustness.

Contribution

We propose a new pipeline connecting scene geometry with multiple sensor signals and introduce CroMo, the first dataset with synchronized multimodal depth data for training and evaluation.

Findings

Outperforms existing monocular depth estimation methods

Effectively combines polarisation, ToF, and structured-light signals

Demonstrates robustness on challenging video scenes

Abstract

Learning-based depth estimation has witnessed recent progress in multiple directions; from self-supervision using monocular video to supervised methods offering highest accuracy. Complementary to supervision, further boosts to performance and robustness are gained by combining information from multiple signals. In this paper we systematically investigate key trade-offs associated with sensor and modality design choices as well as related model training strategies. Our study leads us to a new method, capable of connecting modality-specific advantages from polarisation, Time-of-Flight and structured-light inputs. We propose a novel pipeline capable of estimating depth from monocular polarisation for which we evaluate various training signals. The inversion of differentiable analytic models thereby connects scene geometry with polarisation and ToF signals and enables self-supervised and cross-modal learning. In the absence of existing multimodal datasets, we examine our approach with a custom-made multi-modal camera rig and collect CroMo; the first dataset to consist of synchronized stereo polarisation, indirect ToF and structured-light depth, captured at video rates. Extensive experiments on challenging video scenes confirm both qualitative and quantitative pipeline advantages where we are able to outperform competitive monocular depth estimation method.