A Simple yet Effective Test-Time Adaptation for Zero-Shot Monocular Metric Depth Estimation

TL;DR

This paper introduces a simple, effective method for test-time adaptation of zero-shot monocular depth estimation models that rescales predictions using sparse 3D points, avoiding costly fine-tuning and maintaining model generalization.

Contribution

The proposed method rescales depth predictions using sparse 3D points, eliminating the need for fine-tuning and preserving the model's generalization ability.

Findings

Improves zero-shot monocular depth estimation accuracy.

Achieves results comparable to fine-tuned models.

Demonstrates robustness against noise and calibration errors.

Abstract

The recent development of \emph{foundation models} for monocular depth estimation such as Depth Anything paved the way to zero-shot monocular depth estimation. Since it returns an affine-invariant disparity map, the favored technique to recover the metric depth consists in fine-tuning the model. However, this stage is not straightforward, it can be costly and time-consuming because of the training and the creation of the dataset. The latter must contain images captured by the camera that will be used at test time and the corresponding ground truth. Moreover, the fine-tuning may also degrade the generalizing capacity of the original model. Instead, we propose in this paper a new method to rescale Depth Anything predictions using 3D points provided by sensors or techniques such as low-resolution LiDAR or structure-from-motion with poses given by an IMU. This approach avoids fine-tuning and preserves the generalizing power of the original depth estimation model while being robust to the noise of the sparse depth, of the camera-LiDAR calibration or of the depth model. Our experiments highlight enhancements relative to zero-shot monocular metric depth estimation methods, competitive results compared to fine-tuned approaches and a better robustness than depth completion approaches. Code available at github.com/ENSTA-U2IS-AI/depth-rescaling.