Adaptive Stereo Depth Estimation with Multi-Spectral Images Across All Lighting Conditions

TL;DR

This paper introduces a novel multi-spectral stereo depth estimation framework that combines visible and thermal images, employing a cross-modal feature matching module and degradation masking to improve depth accuracy across all lighting conditions.

Contribution

It presents a new stereo-based multi-spectral depth estimation method with a cross-modal feature matching module and degradation masking, achieving state-of-the-art results under challenging lighting.

Findings

Achieves state-of-the-art performance on MS2 dataset.

Produces high-quality depth maps in diverse lighting conditions.

Effectively handles poor lighting with thermal-based degradation masking.

Abstract

Depth estimation under adverse conditions remains a significant challenge. Recently, multi-spectral depth estimation, which integrates both visible light and thermal images, has shown promise in addressing this issue. However, existing algorithms struggle with precise pixel-level feature matching, limiting their ability to fully exploit geometric constraints across different spectra. To address this, we propose a novel framework incorporating stereo depth estimation to enforce accurate geometric constraints. In particular, we treat the visible light and thermal images as a stereo pair and utilize a Cross-modal Feature Matching (CFM) Module to construct a cost volume for pixel-level matching. To mitigate the effects of poor lighting on stereo matching, we introduce Degradation Masking, which leverages robust monocular thermal depth estimation in degraded regions. Our method achieves state-of-the-art (SOTA) performance on the Multi-Spectral Stereo (MS2) dataset, with qualitative evaluations demonstrating high-quality depth maps under varying lighting conditions.