Leveraging Language for Accelerated Learning of Tool Manipulation

TL;DR

This paper explores how integrating natural language descriptions of tools with meta-learning enables robotic control policies to adapt more quickly to new tools across various manipulation tasks.

Contribution

It introduces a method that combines linguistic information from pre-trained language models with meta-learning to improve tool adaptation in robotic manipulation.

Findings

Linguistic features improve adaptation speed in manipulation tasks.

Meta-learning with language descriptions outperforms baseline methods.

The approach generalizes across multiple tool-related tasks.

Abstract

Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.