Can Language Understand Depth?

TL;DR

This paper introduces DepthCLIP, a zero-shot monocular depth estimation method leveraging CLIP's semantic understanding, surpassing some unsupervised methods without any training.

Contribution

It is the first to adapt CLIP's semantic knowledge to perform zero-shot monocular depth estimation for quantified depth tasks.

Findings

DepthCLIP outperforms existing unsupervised methods.

DepthCLIP approaches the performance of early fully-supervised networks.

First demonstration of zero-shot semantic-to-quantified task transfer.

Abstract

Besides image classification, Contrastive Language-Image Pre-training (CLIP) has accomplished extraordinary success for a wide range of vision tasks, including object-level and 3D space understanding. However, it's still challenging to transfer semantic knowledge learned from CLIP into more intricate tasks of quantified targets, such as depth estimation with geometric information. In this paper, we propose to apply CLIP for zero-shot monocular depth estimation, named DepthCLIP. We found that the patches of the input image could respond to a certain semantic distance token and then be projected to a quantified depth bin for coarse estimation. Without any training, our DepthCLIP surpasses existing unsupervised methods and even approaches the early fully-supervised networks. To our best knowledge, we are the first to conduct zero-shot adaptation from the semantic language knowledge to quantified downstream tasks and perform zero-shot monocular depth estimation. We hope our work could cast a light on future research. The code is available at https://github.com/Adonis-galaxy/DepthCLIP.