INTENTION: Inferring Tendencies of Humanoid Robot Motion Through Interactive Intuition and Grounded VLM

TL;DR

INTENTION is a framework that enables robots to infer and execute appropriate manipulation behaviors in diverse scenarios by integrating vision-language models, scene reasoning, and interaction-driven memory, mimicking human intuitive understanding.

Contribution

The paper introduces a novel framework combining VLMs, scene reasoning, and memory for autonomous, intuitive robot manipulation in unstructured environments.

Findings

Effective scene reasoning with VLMs for manipulation tasks

Memory Graph improves task understanding and decision-making

Robots demonstrate human-like adaptability in diverse scenarios

Abstract

Traditional control and planning for robotic manipulation heavily rely on precise physical models and predefined action sequences. While effective in structured environments, such approaches often fail in real-world scenarios due to modeling inaccuracies and struggle to generalize to novel tasks. In contrast, humans intuitively interact with their surroundings, demonstrating remarkable adaptability, making efficient decisions through implicit physical understanding. In this work, we propose INTENTION, a novel framework enabling robots with learned interactive intuition and autonomous manipulation in diverse scenarios, by integrating Vision-Language Models (VLMs) based scene reasoning with interaction-driven memory. We introduce Memory Graph to record scenes from previous task interactions which embodies human-like understanding and decision-making about different tasks in real world. Meanwhile, we design an Intuitive Perceptor that extracts physical relations and affordances from visual scenes. Together, these components empower robots to infer appropriate interaction behaviors in new scenes without relying on repetitive instructions. Videos: https://robo-intention.github.io