ObjCAViT: Improving Monocular Depth Estimation Using Natural Language Models And Image-Object Cross-Attention

TL;DR

This paper introduces ObjCAViT, a novel approach that leverages language models and cross-attention mechanisms to incorporate semantic and relational scene information, significantly improving monocular depth estimation accuracy.

Contribution

The paper presents a new module, ObjCAViT, that integrates language-based world knowledge and inter-object relationships into monocular depth estimation using transformer attention.

Findings

Achieves highly accurate depth maps on NYUv2 and KITTI datasets.

Using language and cross-attention improves depth estimation performance.

Demonstrates the importance of semantic and relational information in MDE.

Abstract

While monocular depth estimation (MDE) is an important problem in computer vision, it is difficult due to the ambiguity that results from the compression of a 3D scene into only 2 dimensions. It is common practice in the field to treat it as simple image-to-image translation, without consideration for the semantics of the scene and the objects within it. In contrast, humans and animals have been shown to use higher-level information to solve MDE: prior knowledge of the nature of the objects in the scene, their positions and likely configurations relative to one another, and their apparent sizes have all been shown to help resolve this ambiguity. In this paper, we present a novel method to enhance MDE performance by encouraging use of known-useful information about the semantics of objects and inter-object relationships within a scene. Our novel ObjCAViT module sources world-knowledge from language models and learns inter-object relationships in the context of the MDE problem using transformer attention, incorporating apparent size information. Our method produces highly accurate depth maps, and we obtain competitive results on the NYUv2 and KITTI datasets. Our ablation experiments show that the use of language and cross-attention within the ObjCAViT module increases performance. Code is released at https://github.com/DylanAuty/ObjCAViT.