Affordance Perception by a Knowledge-Guided Vision-Language Model with Efficient Error Correction

TL;DR

This paper enhances robot affordance perception in open-world environments by integrating a detailed affordance knowledge base with vision-language models and human-in-the-loop corrections, enabling effective object interaction and task execution.

Contribution

It introduces a precise affordance representation, connects it to vision-language models for unseen objects, and incorporates human feedback for improved perception.

Findings

Effective for robot object search and manipulation

Improves affordance understanding in open-world scenarios

Demonstrated in door opening tasks

Abstract

Mobile robot platforms will increasingly be tasked with activities that involve grasping and manipulating objects in open world environments. Affordance understanding provides a robot with means to realise its goals and execute its tasks, e.g. to achieve autonomous navigation in unknown buildings where it has to find doors and ways to open these. In order to get actionable suggestions, robots need to be able to distinguish subtle differences between objects, as they may result in different action sequences: doorknobs require grasp and twist, while handlebars require grasp and push. In this paper, we improve affordance perception for a robot in an open-world setting. Our contribution is threefold: (1) We provide an affordance representation with precise, actionable affordances; (2) We connect this knowledge base to a foundational vision-language models (VLM) and prompt the VLM for a wider variety of new and unseen objects; (3) We apply a human-in-the-loop for corrections on the output of the VLM. The mix of affordance representation, image detection and a human-in-the-loop is effective for a robot to search for objects to achieve its goals. We have demonstrated this in a scenario of finding various doors and the many different ways to open them.