HybridDepth: Robust Metric Depth Fusion by Leveraging Depth from Focus and Single-Image Priors

TL;DR

HYBRIDDEPTH is a novel depth estimation system that combines focal stack data and single-image priors to produce accurate, robust, and generalizable metric depth maps, suitable for mobile and diverse environments.

Contribution

It introduces a hybrid pipeline leveraging focal stacks and depth priors, improving accuracy, robustness, and zero-shot generalization over existing methods.

Findings

Outperforms state-of-the-art models on DDFF12 and NYU Depth V2 datasets.

Demonstrates strong zero-shot generalization to unseen datasets.

Produces more structurally accurate depth maps on mobile devices.

Abstract

We propose HYBRIDDEPTH, a robust depth estimation pipeline that addresses key challenges in depth estimation,including scale ambiguity, hardware heterogeneity, and generalizability. HYBRIDDEPTH leverages focal stack, data conveniently accessible in common mobile devices, to produce accurate metric depth maps. By incorporating depth priors afforded by recent advances in singleimage depth estimation, our model achieves a higher level of structural detail compared to existing methods. We test our pipeline as an end-to-end system, with a newly developed mobile client to capture focal stacks, which are then sent to a GPU-powered server for depth estimation. Comprehensive quantitative and qualitative analyses demonstrate that HYBRIDDEPTH outperforms state-of-the-art(SOTA) models on common datasets such as DDFF12 and NYU Depth V2. HYBRIDDEPTH also shows strong zero-shot generalization. When trained on NYU Depth V2, HYBRIDDEPTH surpasses SOTA models in zero-shot performance on ARKitScenes and delivers more structurally accurate depth maps on Mobile Depth. The code is available at https://github.com/cake-lab/HybridDepth/.