DeepSight: Bridging Depth Maps and Language with a Depth-Driven Multimodal Model

TL;DR

DeepSight is a novel multimodal large language model specifically designed to improve 3D scene understanding by effectively integrating depth map information with language, utilizing a new dataset and modified encoder architecture.

Contribution

It introduces a dedicated depth-focused multimodal model, new datasets for depth instruction and image-text pairs, and a modified encoder to better capture depth information.

Findings

DeepSight significantly improves depth perception accuracy.

Enhanced performance on depth-related visual question answering.

Effective integration of depth maps boosts 3D scene understanding.

Abstract

Multimodal large language models (MLLMs) have achieved impressive performance across various tasks such as image captioning and visual question answer(VQA); however, they often struggle to accurately interpret depth information inherent in visual data. In this work, we introduce DeepSight, the first dedicated depth MLLM designed to enhance three-dimensional scene understanding. Unlike conventional methods that align RGB image encodings with text, our approach takes advantage of the unique characteristics of depth images: single-channel grayscale images where the pixel values directly reflect depth cues to improve spatial reasoning. To address challenges associated with limited depth data and the inadequacy of simple channel replication, we construct a novel depth image-text pair dataset and a depth instruction dataset. Depth maps are generated from visual images using the GLPN model, and GPT-4 is employed to curate corresponding depth instructions, an approach validated by LLaVA. Additionally, we modify the ViT encoder in CLIP to incorporate local object information, thereby capturing the subtle continuous variations of depth more effectively. To evaluate the performance of our model, we develop a comprehensive depth question answer benchmark based on existing depth image datasets, which rigorously assesses understanding in typical depth map scenarios. Experimental results demonstrate that DeepSight significantly enhances depth perception and downstream task performance, marking a substantial step forward in multimodal three-dimensional understanding.