Gaze-Guided Task Decomposition for Imitation Learning in Robotic Manipulation

TL;DR

This paper introduces a gaze-based method for decomposing robotic manipulation tasks into sub-tasks, improving imitation learning by leveraging human gaze transitions during demonstrations.

Contribution

It presents a simple, robust gaze transition-based task decomposition method that ensures consistent sub-task segmentation across demonstrations in robotic manipulation.

Findings

Method is robust across different hyperparameters.

Ensures consistent task decomposition across demonstrations.

Applicable to various robotic systems.

Abstract

In imitation learning for robotic manipulation, decomposing object manipulation tasks into sub-tasks enables the reuse of learned skills and the combination of learned behaviors to perform novel tasks, rather than simply replicating demonstrated motions. Human gaze is closely linked to hand movements during object manipulation. We hypothesize that an imitating agent's gaze control, fixating on specific landmarks and transitioning between them, simultaneously segments demonstrated manipulations into sub-tasks. This study proposes a simple yet robust task decomposition method based on gaze transitions. Using teleoperation, a common modality in robotic manipulation for collecting demonstrations, in which a human operator's gaze is measured and used for task decomposition as a substitute for an imitating agent's gaze. Our approach ensures consistent task decomposition across all demonstrations for each task, which is desirable in contexts such as machine learning. We evaluated the method across demonstrations of various tasks, assessing the characteristics and consistency of the resulting sub-tasks. Furthermore, extensive testing across different hyperparameter settings confirmed its robustness, making it adaptable to diverse robotic systems. Our code is available at https://github.com/crumbyRobotics/GazeTaskDecomp.