Scene Graph-Guided Proactive Replanning for Failure-Resilient Embodied Agent

TL;DR

This paper introduces a proactive replanning framework for embodied agents that detects scene mismatches early using scene graphs, enabling correction before failures occur, thus enhancing robustness.

Contribution

It proposes a novel proactive replanning method using scene graphs to identify and correct scene mismatches at subtask boundaries, improving autonomous robot robustness.

Findings

Detects scene mismatches before failure

Significantly improves task success rate

Operates with minimal supervision

Abstract

When humans perform everyday tasks, we naturally adjust our actions based on the current state of the environment. For instance, if we intend to put something into a drawer but notice it is closed, we open it first. However, many autonomous robots lack this adaptive awareness. They often follow pre-planned actions that may overlook subtle yet critical changes in the scene, which can result in actions being executed under outdated assumptions and eventual failure. While replanning is critical for robust autonomy, most existing methods respond only after failures occur, when recovery may be inefficient or infeasible. While proactive replanning holds promise for preventing failures in advance, current solutions often rely on manually designed rules and extensive supervision. In this work, we present a proactive replanning framework that detects and corrects failures at subtask boundaries by comparing scene graphs constructed from current RGB-D observations against reference graphs extracted from successful demonstrations. When the current scene fails to align with reference trajectories, a lightweight reasoning module is activated to diagnose the mismatch and adjust the plan. Experiments in the AI2-THOR simulator demonstrate that our approach detects semantic and spatial mismatches before execution failures occur, significantly improving task success and robustness.