Iris: Integrating Language into Diffusion-based Monocular Depth Estimation

TL;DR

This paper introduces Iris, a method that incorporates textual descriptions into diffusion-based monocular depth estimation to improve accuracy and scene understanding, especially in small regions, by leveraging language as an additional condition.

Contribution

The paper presents a novel approach to integrate language into diffusion-based depth estimation models, enhancing their accuracy and interpretability compared to previous purely visual methods.

Findings

Improved depth estimation accuracy, especially in small regions.

Enhanced depth perception of text-described specific regions.

Language acts as a constraint to accelerate training and inference.

Abstract

Traditional monocular depth estimation suffers from inherent ambiguity and visual nuisances. We demonstrate that language can enhance monocular depth estimation by providing an additional condition (rather than images alone) aligned with plausible 3D scenes, thereby reducing the solution space for depth estimation. This conditional distribution is learned during the text-to-image pre-training of diffusion models. To generate images under various viewpoints and layouts that precisely reflect textual descriptions, the model implicitly models object sizes, shapes, and scales, their spatial relationships, and the overall scene structure. In this paper, Iris, we investigate the benefits of our strategy to integrate text descriptions into training and inference of diffusion-based depth estimation models. We experiment with three different diffusion-based monocular depth estimators (Marigold, Lotus, and E2E-FT) and their variants. By training on HyperSim and Virtual KITTI, and evaluating on NYUv2, KITTI, ETH3D, ScanNet, and DIODE, we find that our strategy improves the overall monocular depth estimation accuracy, especially in small areas. It also improves the model's depth perception of specific regions described in the text. We find that by providing more details in the text, the depth prediction can be iteratively refined. Simultaneously, we find that language can act as a constraint to accelerate the convergence of both training and the inference diffusion trajectory. Code and generated text data will be released upon acceptance.