Extending Foundational Monocular Depth Estimators to Fisheye Cameras with Calibration Tokens

TL;DR

This paper introduces a novel calibration token mechanism that allows existing monocular depth estimators trained on perspective images to be effectively adapted for fisheye cameras without retraining, by aligning latent embeddings.

Contribution

The authors propose Calibration Tokens, a lightweight, self-supervised method to adapt FMDEs for fisheye images by aligning latent embeddings, avoiding retraining or fine-tuning.

Findings

Consistently improves depth estimation accuracy on indoor and outdoor datasets.

Requires only large-scale perspective datasets and no fisheye images for training.

Outperforms state-of-the-art methods with a unified token approach.

Abstract

We propose a method to extend foundational monocular depth estimators (FMDEs), trained on perspective images, to fisheye images. Despite being trained on tens of millions of images, FMDEs are susceptible to the covariate shift introduced by changes in camera calibration (intrinsic, distortion) parameters, leading to erroneous depth estimates. Our method aligns the distribution of latent embeddings encoding fisheye images to those of perspective images, enabling the reuse of FMDEs for fisheye cameras without retraining or finetuning. To this end, we introduce a set of Calibration Tokens as a light-weight adaptation mechanism that modulates the latent embeddings for alignment. By exploiting the already expressive latent space of FMDEs, we posit that modulating their embeddings avoids the negative impact of artifacts and loss introduced in conventional recalibration or map projection to a canonical reference frame in the image space. Our method is self-supervised and does not require fisheye images but leverages publicly available large-scale perspective image datasets. This is done by recalibrating perspective images to fisheye images, and enforcing consistency between their estimates during training. We evaluate our approach with several FMDEs, on both indoors and outdoors, where we consistently improve over state-of-the-art methods using a single set of tokens for both. Code available at: https://github.com/JungHeeKim29/calibration-token.