SAGA: Open-World Mobile Manipulation via Structured Affordance Grounding

TL;DR

SAGA introduces a structured affordance grounding framework for mobile manipulation that generalizes across environments and task specifications, enabling versatile visuomotor control with improved performance.

Contribution

The paper presents a novel affordance-based task representation grounded in multimodal foundation models, facilitating generalist mobile manipulation with zero-shot and few-shot capabilities.

Findings

Outperforms end-to-end and modular baselines significantly

Successfully generalizes across 11 real-world tasks

Enables zero-shot and few-shot task execution

Abstract

We present SAGA, a versatile and adaptive framework for visuomotor control that can generalize across various environments, task objectives, and user specifications. To efficiently learn such capability, our key idea is to disentangle high-level semantic intent from low-level visuomotor control by explicitly grounding task objectives in the observed environment. Using an affordance-based task representation, we express diverse and complex behaviors in a unified, structured form. By leveraging multimodal foundation models, SAGA grounds the proposed task representation to the robot's visual observation as 3D affordance heatmaps, highlighting task-relevant entities while abstracting away spurious appearance variations that would hinder generalization. These grounded affordances enable us to effectively train a conditional policy on multi-task demonstration data for whole-body control. In a unified framework, SAGA can solve tasks specified in different forms, including language instructions, selected points, and example demonstrations, enabling both zero-shot execution and few-shot adaptation. We instantiate SAGA on a quadrupedal manipulator and conduct extensive experiments across eleven real-world tasks. SAGA consistently outperforms end-to-end and modular baselines by substantial margins. Together, these results demonstrate that structured affordance grounding offers a scalable and effective pathway toward generalist mobile manipulation.