LLMI3D: MLLM-based 3D Perception from a Single 2D Image

TL;DR

LLMI3D introduces a multimodal large language model tailored for 3D perception from a single 2D image, leveraging novel feature extraction, geometric decoding, and focal variation handling to outperform existing methods.

Contribution

This paper presents LLMI3D, a novel MLLM-based 3D perception model with specialized modules and a new dataset, achieving state-of-the-art results in 3D understanding from 2D images.

Findings

LLMI3D outperforms existing methods significantly.

The IG3D dataset enables detailed 3D perception evaluation.

Parameter-efficient fine-tuning enhances model performance.

Abstract

Recent advancements in autonomous driving, augmented reality, robotics, and embodied intelligence have necessitated 3D perception algorithms. However, current 3D perception methods, especially specialized small models, exhibit poor generalization in open scenarios. On the other hand, multimodal large language models (MLLMs) excel in general capacity but underperform in 3D tasks, due to weak 3D local spatial object perception, poor text-based geometric numerical output, and inability to handle camera focal variations. To address these challenges, we propose the following solutions: Spatial-Enhanced Local Feature Mining for better spatial feature extraction, 3D Query Token-Derived Info Decoding for precise geometric regression, and Geometry Projection-Based 3D Reasoning for handling camera focal length variations. We employ parameter-efficient fine-tuning for a pre-trained MLLM and develop LLMI3D, a powerful 3D perception MLLM. Additionally, we have constructed the IG3D dataset, which provides fine-grained descriptions and question-answer annotations. Extensive experiments demonstrate that our LLMI3D achieves state-of-the-art performance, outperforming other methods by a large margin.