GraphPad: Inference-Time 3D Scene Graph Updates for Embodied Question Answering

TL;DR

GraphPad introduces a dynamic, modifiable 3D scene graph memory for embodied agents, enabling real-time updates and task-specific refinement that improve scene understanding and question answering performance.

Contribution

It presents GraphPad, a novel API-driven, mutable scene graph memory system that adapts during tasks without additional training, enhancing embodied agent capabilities.

Findings

Achieves 55.3% accuracy on OpenEQA, outperforming static baselines.

Operates with five times fewer input frames, reducing computational load.

Enables online, language-driven scene graph updates for better task alignment.

Abstract

Structured scene representations are a core component of embodied agents, helping to consolidate raw sensory streams into readable, modular, and searchable formats. Due to their high computational overhead, many approaches build such representations in advance of the task. However, when the task specifications change, such static approaches become inadequate as they may miss key objects, spatial relations, and details. We introduce GraphPad, a modifiable structured memory that an agent can tailor to the needs of the task through API calls. It comprises a mutable scene graph representing the environment, a navigation log indexing frame-by-frame content, and a scratchpad for task-specific notes. Together, GraphPad serves as a dynamic workspace that remains complete, current, and aligned with the agent's immediate understanding of the scene and its task. On the OpenEQA benchmark, GraphPad attains 55.3%, a +3.0% increase over an image-only baseline using the same vision-language model, while operating with five times fewer input frames. These results show that allowing online, language-driven refinement of 3-D memory yields more informative representations without extra training or data collection.