Learning Articulated Motion Models from Visual and Lingual Signals

TL;DR

This paper introduces a multimodal learning framework that combines visual and lingual signals to accurately model the kinematics of articulated objects, improving robot manipulation capabilities in human environments.

Contribution

It presents a novel probabilistic language model linking natural language descriptions with kinematic structures, enhancing model inference over vision-only methods.

Findings

36% improvement in model accuracy over vision-only baseline

Successfully infers kinematic structures of complex household objects

Demonstrates the effectiveness of multimodal signals in robotic manipulation

Abstract

In order for robots to operate effectively in homes and workplaces, they must be able to manipulate the articulated objects common within environments built for and by humans. Previous work learns kinematic models that prescribe this manipulation from visual demonstrations. Lingual signals, such as natural language descriptions and instructions, offer a complementary means of conveying knowledge of such manipulation models and are suitable to a wide range of interactions (e.g., remote manipulation). In this paper, we present a multimodal learning framework that incorporates both visual and lingual information to estimate the structure and parameters that define kinematic models of articulated objects. The visual signal takes the form of an RGB-D image stream that opportunistically captures object motion in an unprepared scene. Accompanying natural language descriptions of the motion constitute the lingual signal. We present a probabilistic language model that uses word embeddings to associate lingual verbs with their corresponding kinematic structures. By exploiting the complementary nature of the visual and lingual input, our method infers correct kinematic structures for various multiple-part objects on which the previous state-of-the-art, visual-only system fails. We evaluate our multimodal learning framework on a dataset comprised of a variety of household objects, and demonstrate a 36% improvement in model accuracy over the vision-only baseline.