Scaling 3D Reasoning with LMMs to Large Robot Mission Environments Using Datagraphs

TL;DR

This paper introduces a datagraph-based method to enable Large Multimodal Models to efficiently handle large 3D environments, improving scalability for robot search-and-rescue missions.

Contribution

It presents a novel datagraph structure combined with graph traversal algorithms to scale LMMs to large 3D scenes, addressing input size limitations.

Findings

Datagraph allows iterative querying of large environments.

Prioritized scene traversal enhances relevance and efficiency.

Method demonstrated on search-and-rescue 3D scene tasks.

Abstract

This paper addresses the challenge of scaling Large Multimodal Models (LMMs) to expansive 3D environments. Solving this open problem is especially relevant for robot deployment in many first-responder scenarios, such as search-and-rescue missions that cover vast spaces. The use of LMMs in these settings is currently hampered by the strict context windows that limit the LMM's input size. We therefore introduce a novel approach that utilizes a datagraph structure, which allows the LMM to iteratively query smaller sections of a large environment. Using the datagraph in conjunction with graph traversal algorithms, we can prioritize the most relevant locations to the query, thereby improving the scalability of 3D scene language tasks. We illustrate the datagraph using 3D scenes, but these can be easily substituted by other dense modalities that represent the environment, such as pointclouds or Gaussian splats. We demonstrate the potential to use the datagraph for two 3D scene language task use cases, in a search-and-rescue mission example.