Context Matters! Relaxing Goals with LLMs for Feasible 3D Scene Planning

TL;DR

This paper introduces ContextMatters, a framework that combines LLMs and classical planning to adapt goals in 3D environments, improving feasibility and success rates in robotic scene planning.

Contribution

It presents a hierarchical goal relaxation method that fuses LLMs with classical planning, enabling context-aware goal adaptation in 3D scene planning.

Findings

+52.45% success rate over baseline

Effective goal relaxation in complex 3D scenes

Validated on real-world robot (TIAGo)

Abstract

Embodied agents need to plan and act reliably in real and complex 3D environments. Classical planning (e.g., PDDL) offers structure and guarantees, but in practice it fails under noisy perception and incorrect predicate grounding. On the other hand, Large Language Models (LLMs)-based planners leverage commonsense reasoning, yet frequently propose actions that are unfeasible or unsafe. Following recent works that combine the two approaches, we introduce ContextMatters, a framework that fuses LLMs and classical planning to perform hierarchical goal relaxation: the LLM helps ground symbols to the scene and, when the target is unreachable, it proposes functionally equivalent goals that progressively relax constraints, adapting the goal to the context of the agent's environment. Operating on 3D Scene Graphs, this mechanism turns many nominally unfeasible tasks into tractable plans and enables context-aware partial achievement when full completion is not achievable. Our experimental results show a +52.45% Success Rate improvement over state-of-the-art LLMs+PDDL baseline, demonstrating the effectiveness of our approach. Moreover, we validate the execution of ContextMatter in a real world scenario by deploying it on a TIAGo robot. Code, dataset, and supplementary materials are available to the community at https://lab-rococo-sapienza.github.io/context-matters/.