MomaGraph: State-Aware Unified Scene Graphs with Vision-Language Model for Embodied Task Planning

TL;DR

MomaGraph introduces a unified, task-driven scene graph representation for embodied agents, supported by a large dataset and a vision-language model that excels in zero-shot planning and understanding in household environments.

Contribution

The paper presents MomaGraph, a novel unified scene graph model with a large dataset and evaluation suite, enabling improved embodied task planning and scene understanding.

Findings

Achieved 71.6% accuracy on MomaGraph-Bench, surpassing baselines by 11.4%.

Demonstrated effective zero-shot task planning and generalization to real robots.

Provided the first large-scale, task-driven scene graph dataset for household environments.

Abstract

Mobile manipulators in households must both navigate and manipulate. This requires a compact, semantically rich scene representation that captures where objects are, how they function, and which parts are actionable. Scene graphs are a natural choice, yet prior work often separates spatial and functional relations, treats scenes as static snapshots without object states or temporal updates, and overlooks information most relevant for accomplishing the current task. To address these limitations, we introduce MomaGraph, a unified scene representation for embodied agents that integrates spatial-functional relationships and part-level interactive elements. However, advancing such a representation requires both suitable data and rigorous evaluation, which have been largely missing. We thus contribute MomaGraph-Scenes, the first large-scale dataset of richly annotated, task-driven scene graphs in household environments, along with MomaGraph-Bench, a systematic evaluation suite spanning six reasoning capabilities from high-level planning to fine-grained scene understanding. Built upon this foundation, we further develop MomaGraph-R1, a 7B vision-language model trained with reinforcement learning on MomaGraph-Scenes. MomaGraph-R1 predicts task-oriented scene graphs and serves as a zero-shot task planner under a Graph-then-Plan framework. Extensive experiments demonstrate that our model achieves state-of-the-art results among open-source models, reaching 71.6% accuracy on the benchmark (+11.4% over the best baseline), while generalizing across public benchmarks and transferring effectively to real-robot experiments.