Learning Agent-Aware Affordances for Closed-Loop Interaction with Articulated Objects

TL;DR

This paper presents a novel closed-loop control pipeline for mobile robots to interact with articulated objects, leveraging agent-aware affordances that improve task success and robustness in real-world scenarios.

Contribution

It introduces agent-aware affordances that account for the robot's capabilities, enhancing manipulation performance over traditional end-effector conditioned methods.

Findings

Agent-aware affordances outperform state-of-the-art methods.

Closed-loop inference enables task segmentation and recovery.

High success rates in real-world long-horizon tasks (opening: 71%, closing: 72%).

Abstract

Interactions with articulated objects are a challenging but important task for mobile robots. To tackle this challenge, we propose a novel closed-loop control pipeline, which integrates manipulation priors from affordance estimation with sampling-based whole-body control. We introduce the concept of agent-aware affordances which fully reflect the agent's capabilities and embodiment and we show that they outperform their state-of-the-art counterparts which are only conditioned on the end-effector geometry. Additionally, closed-loop affordance inference is found to allow the agent to divide a task into multiple non-continuous motions and recover from failure and unexpected states. Finally, the pipeline is able to perform long-horizon mobile manipulation tasks, i.e. opening and closing an oven, in the real world with high success rates (opening: 71%, closing: 72%).