Rethinking Intermediate Representation for VLM-based Robot Manipulation

TL;DR

This paper introduces SEAM, a semantic assembly representation for VLM-based robot manipulation that balances comprehensibility and generalizability, improving task handling and inference speed.

Contribution

The paper proposes SEAM, a grammar-based intermediate representation, and a novel segmentation paradigm with retrieval-augmented few-shot learning for enhanced robot manipulation.

Findings

SEAM outperforms existing representations in generalizability and comprehensibility metrics.

The new segmentation method achieves the shortest inference time among state-of-the-art approaches.

Real-world experiments demonstrate SOTA performance across various tasks.

Abstract

Vision-Language Model (VLM) is an important component to enable robust robot manipulation. Yet, using it to translate human instructions into an action-resolvable intermediate representation often needs a tradeoff between VLM-comprehensibility and generalizability. Inspired by context-free grammar, we design the Semantic Assembly representation named SEAM, by decomposing the intermediate representation into vocabulary and grammar. Doing so leads us to a concise vocabulary of semantically-rich operations and a VLM-friendly grammar for handling diverse unseen tasks. In addition, we design a new open-vocabulary segmentation paradigm with a retrieval-augmented few-shot learning strategy to localize fine-grained object parts for manipulation, effectively with the shortest inference time over all state-of-the-art parallel works. Also, we formulate new metrics for action-generalizability and VLM-comprehensibility, demonstrating the compelling performance of SEAM over mainstream representations on both aspects. Extensive real-world experiments further manifest its SOTA performance under varying settings and tasks.