Which objects help me to act effectively? Reasoning about physically-grounded affordances

TL;DR

This paper presents a method combining large language models and vision-language models to detect affordances of objects in open-world environments, considering physical properties and robot embodiment for effective interaction.

Contribution

It introduces a novel approach that grounds affordance detection in physical properties and robot embodiment using LLMs and VLMs, enabling open-vocabulary and context-aware object interaction.

Findings

Method successfully identifies useful objects among distractors.

Finetuning VLMs enhances physical property understanding.

Grounding in physical properties improves affordance detection accuracy.

Abstract

For effective interactions with the open world, robots should understand how interactions with known and novel objects help them towards their goal. A key aspect of this understanding lies in detecting an object's affordances, which represent the potential effects that can be achieved by manipulating the object in various ways. Our approach leverages a dialogue of large language models (LLMs) and vision-language models (VLMs) to achieve open-world affordance detection. Given open-vocabulary descriptions of intended actions and effects, the useful objects in the environment are found. By grounding our system in the physical world, we account for the robot's embodiment and the intrinsic properties of the objects it encounters. In our experiments, we have shown that our method produces tailored outputs based on different embodiments or intended effects. The method was able to select a useful object from a set of distractors. Finetuning the VLM for physical properties improved overall performance. These results underline the importance of grounding the affordance search in the physical world, by taking into account robot embodiment and the physical properties of objects.