ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth

TL;DR

ZoeDepth introduces a novel depth estimation model that combines relative and metric depth learning, achieving superior generalization and state-of-the-art results across multiple datasets and domains.

Contribution

It is the first model to integrate relative and metric depth training, enabling zero-shot generalization and improved accuracy on diverse datasets.

Findings

Significantly improves SOTA on NYU Depth v2 without pre-training.

Achieves 21% improvement in REL with pre-training on 12 datasets.

Demonstrates strong zero-shot generalization to unseen datasets.

Abstract

This paper tackles the problem of depth estimation from a single image. Existing work either focuses on generalization performance disregarding metric scale, i.e. relative depth estimation, or state-of-the-art results on specific datasets, i.e. metric depth estimation. We propose the first approach that combines both worlds, leading to a model with excellent generalization performance while maintaining metric scale. Our flagship model, ZoeD-M12-NK, is pre-trained on 12 datasets using relative depth and fine-tuned on two datasets using metric depth. We use a lightweight head with a novel bin adjustment design called metric bins module for each domain. During inference, each input image is automatically routed to the appropriate head using a latent classifier. Our framework admits multiple configurations depending on the datasets used for relative depth pre-training and metric fine-tuning. Without pre-training, we can already significantly improve the state of the art (SOTA) on the NYU Depth v2 indoor dataset. Pre-training on twelve datasets and fine-tuning on the NYU Depth v2 indoor dataset, we can further improve SOTA for a total of 21% in terms of relative absolute error (REL). Finally, ZoeD-M12-NK is the first model that can jointly train on multiple datasets (NYU Depth v2 and KITTI) without a significant drop in performance and achieve unprecedented zero-shot generalization performance to eight unseen datasets from both indoor and outdoor domains. The code and pre-trained models are publicly available at https://github.com/isl-org/ZoeDepth .