RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulation

TL;DR

This paper presents RoboEXP, a system for autonomous robotic scene exploration that constructs an action-conditioned scene graph integrating geometric, semantic, and relational information to improve manipulation tasks.

Contribution

The paper introduces the novel task of interactive scene exploration and develops RoboEXP, combining multimodal reasoning and memory to build detailed scene graphs for robotic manipulation.

Findings

RoboEXP effectively constructs comprehensive scene graphs during exploration.

The system improves manipulation efficiency across various object types.

Experimental results demonstrate robustness and versatility in real-world scenarios.

Abstract

We introduce the novel task of interactive scene exploration, wherein robots autonomously explore environments and produce an action-conditioned scene graph (ACSG) that captures the structure of the underlying environment. The ACSG accounts for both low-level information (geometry and semantics) and high-level information (action-conditioned relationships between different entities) in the scene. To this end, we present the Robotic Exploration (RoboEXP) system, which incorporates the Large Multimodal Model (LMM) and an explicit memory design to enhance our system's capabilities. The robot reasons about what and how to explore an object, accumulating new information through the interaction process and incrementally constructing the ACSG. Leveraging the constructed ACSG, we illustrate the effectiveness and efficiency of our RoboEXP system in facilitating a wide range of real-world manipulation tasks involving rigid, articulated objects, nested objects, and deformable objects.