Cybo-Waiter: A Physical Agentic Framework for Humanoid Whole-Body Locomotion-Manipulation

TL;DR

This paper introduces Cybo-Waiter, a humanoid framework that integrates visual language planning, 3D object grounding, and coordinated locomotion-manipulation to improve robustness in complex tasks.

Contribution

It presents a novel framework that converts language plans into verifiable task programs and coordinates humanoid actions with multi-object grounding and replanning capabilities.

Findings

Enhanced robustness in humanoid manipulation tasks

Effective multi-object grounding and diagnostics

Successful long-horizon loco-manipulation experiments

Abstract

Robots are increasingly expected to execute open ended natural language requests in human environments, which demands reliable long horizon execution under partial observability. This is especially challenging for humanoids because locomotion and manipulation are tightly coupled through stance, reachability, and balance. We present a humanoid agent framework that turns VLM plans into verifiable task programs and closes the loop with multi object 3D geometric supervision. A VLM planner compiles each instruction into a typed JSON sequence of subtasks with explicit predicate based preconditions and success conditions. Using SAM3 and RGB-D, we ground all task relevant entities in 3D, estimate object centroids and extents, and evaluate predicates over stable frames to obtain condition level diagnostics. The supervisor uses these diagnostics to verify subtask completion and to provide condition-level feedback for progression and replanning. We execute each subtask by coordinating humanoid locomotion and whole-body manipulation, selecting feasible motion primitives under reachability and balance constraints. Experiments on tabletop manipulation and long horizon humanoid loco manipulation tasks show improved robustness from multi object grounding, temporal stability, and recovery driven replanning.