MuST: Multi-Head Skill Transformer for Long-Horizon Dexterous Manipulation with Skill Progress

TL;DR

MuST is a novel transformer-based framework that enables robots to learn, sequence, and execute complex long-horizon manipulation tasks by chaining multiple skills with progress-guided transitions.

Contribution

It introduces a multi-head skill transformer with progress values for effective skill chaining and expansion in dexterous manipulation tasks.

Findings

Significantly improves long-horizon manipulation performance

Effective skill sequencing and transition management

Validated in both simulated and real-world environments

Abstract

Robot picking and packing tasks require dexterous manipulation skills, such as rearranging objects to establish a good grasping pose, or placing and pushing items to achieve tight packing. These tasks are challenging for robots due to the complexity and variability of the required actions. To tackle the difficulty of learning and executing long-horizon tasks, we propose a novel framework called the Multi-Head Skill Transformer (MuST). This model is designed to learn and sequentially chain together multiple motion primitives (skills), enabling robots to perform complex sequences of actions effectively. MuST introduces a "progress value" for each skill, guiding the robot on which skill to execute next and ensuring smooth transitions between skills. Additionally, our model is capable of expanding its skill set and managing various sequences of sub-tasks efficiently. Extensive experiments in both simulated and real-world environments demonstrate that MuST significantly enhances the robot's ability to perform long-horizon dexterous manipulation tasks.