Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions

TL;DR

This paper introduces AIDE, a dual-stream framework that enhances robot understanding and execution of ambiguous instructions through interactive exploration and vision-language reasoning, achieving high success and accuracy rates.

Contribution

AIDE integrates interactive exploration with vision-language reasoning, enabling zero-shot affordance analysis and improved real-time decision-making for ambiguous instructions.

Findings

Achieves over 80% task planning success rate.

Attains more than 95% accuracy in continuous execution.

Outperforms existing VLM-based methods in diverse scenarios.

Abstract

Enabling robots to explore and act in unfamiliar environments under ambiguous human instructions by interactively identifying task-relevant objects (e.g., identifying cups or beverages for "I'm thirsty") remains challenging for existing vision-language model (VLM)-based methods. This challenge stems from inefficient reasoning and the lack of environmental interaction, which hinder real-time task planning and execution. To address this, We propose Affordance-Aware Interactive Decision-Making and Execution for Ambiguous Instructions (AIDE), a dual-stream framework that integrates interactive exploration with vision-language reasoning, where Multi-Stage Inference (MSI) serves as the decision-making stream and Accelerated Decision-Making (ADM) as the execution stream, enabling zero-shot affordance analysis and interpretation of ambiguous instructions. Extensive experiments in simulation and real-world environments show that AIDE achieves the task planning success rate of over 80\% and more than 95\% accuracy in closed-loop continuous execution at 10 Hz, outperforming existing VLM-based methods in diverse open-world scenarios.