GaussExplorer: 3D Gaussian Splatting for Embodied Exploration and Reasoning

TL;DR

GaussExplorer combines 3D Gaussian Splatting with vision-language models to enable question-driven exploration and reasoning in 3D scenes, overcoming limitations of prior methods in handling complex queries.

Contribution

It introduces a novel framework that integrates VLMs with 3D Gaussian Splatting for improved embodied exploration and reasoning in complex 3D environments.

Findings

Outperforms existing methods on multiple benchmarks.

Effectively interprets complex, compositional language queries.

Enhances visual reasoning by adjusting viewpoints based on query relevance.

Abstract

We present GaussExplorer, a framework for embodied exploration and reasoning built on 3D Gaussian Splatting (3DGS). While prior approaches to language-embedded 3DGS have made meaningful progress in aligning simple text queries with Gaussian embeddings, they are generally optimized for relatively simple queries and struggle to interpret more complex, compositional language queries. Alternative studies based on object-centric RGB-D structured memories provide spatial grounding but are constrained by pre-fixed viewpoints. To address these issues, GaussExplorer introduces Vision-Language Models (VLMs) on top of 3DGS to enable question-driven exploration and reasoning within 3D scenes. We first identify pre-captured images that are most correlated with the query question, and subsequently adjust them into novel viewpoints to more accurately capture visual information for better reasoning by VLMs. Experiments show that ours outperforms existing methods on several benchmarks, demonstrating the effectiveness of integrating VLM-based reasoning with 3DGS for embodied tasks.