Towards Robust Monocular Depth Estimation in Non-Lambertian Surfaces

TL;DR

This paper introduces a novel training framework for monocular depth estimation that improves robustness on non-Lambertian surfaces by guiding predictions in the gradient domain, employing lighting augmentation, and using a variational autoencoder-based fusion module.

Contribution

It proposes non-Lambertian surface regional guidance, lighting augmentation, and a variational autoencoder fusion module to enhance depth estimation on reflective surfaces.

Findings

33.39% accuracy improvement on Booster dataset

5.21% accuracy improvement on Mirror3D dataset

State-of-the-art 90.75 delta1.05 on TRICKY2024

Abstract

In the field of monocular depth estimation (MDE), many models with excellent zero-shot performance in general scenes emerge recently. However, these methods often fail in predicting non-Lambertian surfaces, such as transparent or mirror (ToM) surfaces, due to the unique reflective properties of these regions. Previous methods utilize externally provided ToM masks and aim to obtain correct depth maps through direct in-painting of RGB images. These methods highly depend on the accuracy of additional input masks, and the use of random colors during in-painting makes them insufficiently robust. We are committed to incrementally enabling the baseline model to directly learn the uniqueness of non-Lambertian surface regions for depth estimation through a well-designed training framework. Therefore, we propose non-Lambertian surface regional guidance, which constrains the predictions of MDE model from the gradient domain to enhance its robustness. Noting the significant impact of lighting on this task, we employ the random tone-mapping augmentation during training to ensure the network can predict correct results for varying lighting inputs. Additionally, we propose an optional novel lighting fusion module, which uses Variational Autoencoders to fuse multiple images and obtain the most advantageous input RGB image for depth estimation when multi-exposure images are available. Our method achieves accuracy improvements of 33.39% and 5.21% in zero-shot testing on the Booster and Mirror3D dataset for non-Lambertian surfaces, respectively, compared to the Depth Anything V2. The state-of-the-art performance of 90.75 in delta1.05 within the ToM regions on the TRICKY2024 competition test set demonstrates the effectiveness of our approach.