Enhancing Reusability of Learned Skills for Robot Manipulation via Gaze Information and Motion Bottlenecks

TL;DR

This paper introduces GazeBot, a novel algorithm that enhances the reusability of learned robot manipulation skills by utilizing gaze information and motion bottlenecks, improving generalization to unseen scenarios.

Contribution

GazeBot is a new gaze-based, bottleneck-aware algorithm that significantly improves skill reusability and generalization in robot manipulation tasks.

Findings

GazeBot achieves higher success rates than state-of-the-art methods.

It generalizes well to unseen object positions and poses.

Training is fully data-driven with demonstration and gaze data.

Abstract

Autonomous agents capable of diverse object manipulations should be able to acquire a wide range of manipulation skills with high reusability. Although advances in deep learning have made it increasingly feasible to replicate the dexterity of human teleoperation in robots, generalizing these acquired skills to previously unseen scenarios remains a significant challenge. In this study, we propose a novel algorithm, Gaze-based Bottleneck-aware Robot Manipulation (GazeBot), which enables high reusability of learned motions without sacrificing dexterity or reactivity. By leveraging gaze information and motion bottlenecks, both crucial features for object manipulation, GazeBot achieves high success rates compared with state-of-the-art imitation learning methods, particularly when the object positions and end-effector poses differ from those in the provided demonstrations. Furthermore, the training process of GazeBot is entirely data-driven once a demonstration dataset with gaze data is provided. Videos and code are available at https://crumbyrobotics.github.io/gazebot.