Depth Any Camera: Zero-Shot Metric Depth Estimation from Any Camera

TL;DR

Depth Any Camera (DAC) is a zero-shot depth estimation framework that generalizes from perspective images to fisheye and 360-degree cameras using a unified ERP representation and innovative augmentation techniques.

Contribution

DAC introduces a novel approach that enables zero-shot metric depth estimation across diverse camera types without specialized training data.

Findings

DAC improves $$ accuracy by up to 50% on fisheye and 360-degree datasets.

DAC generalizes seamlessly from perspective to wide FoV cameras.

State-of-the-art performance in zero-shot depth estimation across multiple camera types.

Abstract

While recent depth foundation models exhibit strong zero-shot generalization, achieving accurate metric depth across diverse camera types-particularly those with large fields of view (FoV) such as fisheye and 360-degree cameras-remains a significant challenge. This paper presents Depth Any Camera (DAC), a powerful zero-shot metric depth estimation framework that extends a perspective-trained model to effectively handle cameras with varying FoVs. The framework is designed to ensure that all existing 3D data can be leveraged, regardless of the specific camera types used in new applications. Remarkably, DAC is trained exclusively on perspective images but generalizes seamlessly to fisheye and 360-degree cameras without the need for specialized training data. DAC employs Equi-Rectangular Projection (ERP) as a unified image representation, enabling consistent processing of images with diverse FoVs. Its core components include pitch-aware Image-to-ERP conversion with efficient online augmentation to simulate distorted ERP patches from undistorted inputs, FoV alignment operations to enable effective training across a wide range of FoVs, and multi-resolution data augmentation to further address resolution disparities between training and testing. DAC achieves state-of-the-art zero-shot metric depth estimation, improving $\delta_1$ accuracy by up to 50% on multiple fisheye and 360-degree datasets compared to prior metric depth foundation models, demonstrating robust generalization across camera types.