Few-shot transfer of tool-use skills using human demonstrations with proximity and tactile sensing

TL;DR

This paper presents a few-shot transfer learning framework for robot tool-use skills using multimodal sensing, combining simulation pre-training and real-world fine-tuning with human demonstrations to overcome data limitations and sim-to-real gaps.

Contribution

It introduces a novel framework that leverages pre-training in simulation and minimal real-world demonstrations to enable robots to learn diverse tool-use skills.

Findings

Effective transfer of tool-use skills with few demonstrations

Enhanced contact state recognition using multimodal sensors

Successful surface-following tasks on diverse tools

Abstract

Tools extend the manipulation abilities of robots, much like they do for humans. Despite human expertise in tool manipulation, teaching robots these skills faces challenges. The complexity arises from the interplay of two simultaneous points of contact: one between the robot and the tool, and another between the tool and the environment. Tactile and proximity sensors play a crucial role in identifying these complex contacts. However, learning tool manipulation using these sensors remains challenging due to limited real-world data and the large sim-to-real gap. To address this, we propose a few-shot tool-use skill transfer framework using multimodal sensing. The framework involves pre-training the base policy to capture contact states common in tool-use skills in simulation and fine-tuning it with human demonstrations collected in the real-world target domain to bridge the domain gap. We validate that this framework enables teaching surface-following tasks using tools with diverse physical and geometric properties with a small number of demonstrations on the Franka Emika robot arm. Our analysis suggests that the robot acquires new tool-use skills by transferring the ability to recognise tool-environment contact relationships from pre-trained to fine-tuned policies. Additionally, combining proximity and tactile sensors enhances the identification of contact states and environmental geometry.