Goal-Oriented Skill Abstraction for Offline Multi-Task Reinforcement Learning

TL;DR

This paper introduces GO-Skill, a goal-oriented skill abstraction method for offline multi-task reinforcement learning, which extracts reusable skills to improve knowledge transfer and task performance without online interaction.

Contribution

It proposes a novel goal-oriented skill extraction and hierarchical policy framework that enhances multi-task learning by leveraging a discrete skill library and skill refinement.

Findings

Effective skill transfer across tasks demonstrated on MetaWorld benchmark

Improved multi-task performance with hierarchical skill composition

Skill refinement phase enhances the usefulness of extracted skills

Abstract

Offline multi-task reinforcement learning aims to learn a unified policy capable of solving multiple tasks using only pre-collected task-mixed datasets, without requiring any online interaction with the environment. However, it faces significant challenges in effectively sharing knowledge across tasks. Inspired by the efficient knowledge abstraction observed in human learning, we propose Goal-Oriented Skill Abstraction (GO-Skill), a novel approach designed to extract and utilize reusable skills to enhance knowledge transfer and task performance. Our approach uncovers reusable skills through a goal-oriented skill extraction process and leverages vector quantization to construct a discrete skill library. To mitigate class imbalances between broadly applicable and task-specific skills, we introduce a skill enhancement phase to refine the extracted skills. Furthermore, we integrate these skills using hierarchical policy learning, enabling the construction of a high-level policy that dynamically orchestrates discrete skills to accomplish specific tasks. Extensive experiments on diverse robotic manipulation tasks within the MetaWorld benchmark demonstrate the effectiveness and versatility of GO-Skill.