Reasoning with Scene Graphs for Robot Planning under Partial Observability

TL;DR

This paper introduces SARP, an algorithm that uses scene graphs derived from visual data to improve robot planning under partial observability, enhancing efficiency and accuracy in complex environments.

Contribution

SARP is a novel method that constructs and reasons with scene graphs from images to enable context-aware planning in uncertain, object-rich domains.

Findings

SARP outperforms standard methods in simulation and real robot experiments.

SARP improves task efficiency and accuracy in target search tasks.

Scene graphs effectively encode contextual information for planning under uncertainty.

Abstract

Robot planning in partially observable domains is difficult, because a robot needs to estimate the current state and plan actions at the same time. When the domain includes many objects, reasoning about the objects and their relationships makes robot planning even more difficult. In this paper, we develop an algorithm called scene analysis for robot planning (SARP) that enables robots to reason with visual contextual information toward achieving long-term goals under uncertainty. SARP constructs scene graphs, a factored representation of objects and their relations, using images captured from different positions, and reasons with them to enable context-aware robot planning under partial observability. Experiments have been conducted using multiple 3D environments in simulation, and a dataset collected by a real robot. In comparison to standard robot planning and scene analysis methods, in a target search domain, SARP improves both efficiency and accuracy in task completion. Supplementary material can be found at https://tinyurl.com/sarp22