Toward Accurate Long-Horizon Robotic Manipulation: Language-to-Action with Foundation Models via Scene Graphs

TL;DR

This paper introduces a framework that uses pre-trained foundation models and scene graphs to enable accurate long-horizon robotic manipulation without domain-specific training, demonstrating promising experimental results.

Contribution

It presents a novel integration of foundation models with scene graphs for robotic manipulation, eliminating the need for domain-specific training data.

Findings

Effective perception and reasoning in manipulation tasks

Robust task sequencing demonstrated in experiments

Potential for building manipulation systems on off-the-shelf models

Abstract

This paper presents a framework that leverages pre-trained foundation models for robotic manipulation without domain-specific training. The framework integrates off-the-shelf models, combining multimodal perception from foundation models with a general-purpose reasoning model capable of robust task sequencing. Scene graphs, dynamically maintained within the framework, provide spatial awareness and enable consistent reasoning about the environment. The framework is evaluated through a series of tabletop robotic manipulation experiments, and the results highlight its potential for building robotic manipulation systems directly on top of off-the-shelf foundation models.