Maestro: Orchestrating Robotics Modules with Vision-Language Models for Zero-Shot Generalist Robots

TL;DR

Maestro leverages vision-language models to dynamically compose robot modules into adaptable policies, achieving superior zero-shot manipulation performance and easy extensibility for diverse robotic embodiments.

Contribution

It introduces a modular framework that integrates VLMs with robot-specific modules, enabling flexible, zero-shot generalist robot behaviors without extensive dataset training.

Findings

Outperforms existing VLA models in zero-shot manipulation tasks.

Easily adaptable to new robot embodiments and modules.

Requires minimal real-world data for adaptation.

Abstract

Today's best-explored routes towards generalist robots center on collecting ever larger "observations-in actions-out" robotics datasets to train large end-to-end models, copying a recipe that has worked for vision-language models (VLMs). We pursue a road less traveled: building generalist policies directly around VLMs by augmenting their general capabilities with specific robot capabilities encapsulated in a carefully curated set of perception, planning, and control modules. In Maestro, a VLM coding agent dynamically composes these modules into a programmatic policy for the current task and scenario. Maestro's architecture benefits from a streamlined closed-loop interface without many manually imposed structural constraints, and a comprehensive and diverse tool repertoire. As a result, it largely surpasses today's VLA models for zero-shot performance on challenging manipulation skills. Further, Maestro is easily extensible to incorporate new modules, easily editable to suit new embodiments such as a quadruped-mounted arm, and even easily adapts from minimal real-world experiences through local code edits.