Lexicon3D: Probing Visual Foundation Models for Complex 3D Scene Understanding

TL;DR

This paper systematically evaluates various visual foundation models for complex 3D scene understanding, revealing their strengths and limitations across multiple tasks and scenarios to guide future model selection.

Contribution

It provides a comprehensive comparison of seven vision encoders across four scene understanding tasks, highlighting key performance insights and challenging existing assumptions.

Findings

DINOv2 outperforms other models in overall performance

Video models excel in object-level scene understanding

Diffusion models enhance geometric task performance

Abstract

Complex 3D scene understanding has gained increasing attention, with scene encoding strategies playing a crucial role in this success. However, the optimal scene encoding strategies for various scenarios remain unclear, particularly compared to their image-based counterparts. To address this issue, we present a comprehensive study that probes various visual encoding models for 3D scene understanding, identifying the strengths and limitations of each model across different scenarios. Our evaluation spans seven vision foundation encoders, including image-based, video-based, and 3D foundation models. We evaluate these models in four tasks: Vision-Language Scene Reasoning, Visual Grounding, Segmentation, and Registration, each focusing on different aspects of scene understanding. Our evaluations yield key findings: DINOv2 demonstrates superior performance, video models excel in object-level tasks, diffusion models benefit geometric tasks, and language-pretrained models show unexpected limitations in language-related tasks. These insights challenge some conventional understandings, provide novel perspectives on leveraging visual foundation models, and highlight the need for more flexible encoder selection in future vision-language and scene-understanding tasks. Code: https://github.com/YunzeMan/Lexicon3D