Words2Contact: Identifying Support Contacts from Verbal Instructions Using Foundation Models

TL;DR

Words2Contact leverages large language and vision models to interpret verbal instructions for robot contact placement, enabling intuitive human-robot interaction and iterative correction in real-world tasks.

Contribution

Introduces a novel language-guided contact placement pipeline that integrates LLMs and VLMs for improved human-robot cooperation and teleoperation.

Findings

Effective iterative correction process for contact placement

Naive users quickly learn to instruct system accurately

Successful real-world validation with humanoid robot Talos

Abstract

This paper presents Words2Contact, a language-guided multi-contact placement pipeline leveraging large language models and vision language models. Our method is a key component for language-assisted teleoperation and human-robot cooperation, where human operators can instruct the robots where to place their support contacts before whole-body reaching or manipulation using natural language. Words2Contact transforms the verbal instructions of a human operator into contact placement predictions; it also deals with iterative corrections, until the human is satisfied with the contact location identified in the robot's field of view. We benchmark state-of-the-art LLMs and VLMs for size and performance in contact prediction. We demonstrate the effectiveness of the iterative correction process, showing that users, even naive, quickly learn how to instruct the system to obtain accurate locations. Finally, we validate Words2Contact in real-world experiments with the Talos humanoid robot, instructed by human operators to place support contacts on different locations and surfaces to avoid falling when reaching for distant objects.