Learning a Skill-sequence-dependent Policy for Long-horizon Manipulation Tasks

TL;DR

This paper introduces a hierarchical reinforcement learning approach that incorporates skill sequences into the policy, significantly improving sample efficiency and performance in long-horizon robotic manipulation tasks.

Contribution

The paper proposes a novel skill-sequence-dependent hierarchical policy that combines high-level skill planning with low-level observation-based control for long-horizon tasks.

Findings

Successfully solves long-horizon manipulation tasks in simulation.

Achieves faster learning compared to PPO and task schema methods.

Enhances sample efficiency by integrating skill sequences into policy learning.

Abstract

In recent years, the robotics community has made substantial progress in robotic manipulation using deep reinforcement learning (RL). Effectively learning of long-horizon tasks remains a challenging topic. Typical RL-based methods approximate long-horizon tasks as Markov decision processes and only consider current observation (images or other sensor information) as input state. However, such approximation ignores the fact that skill-sequence also plays a crucial role in long-horizon tasks. In this paper, we take both the observation and skill sequences into account and propose a skill-sequence-dependent hierarchical policy for solving a typical long-horizon task. The proposed policy consists of a high-level skill policy (utilizing skill sequences) and a low-level parameter policy (responding to observation) with corresponding training methods, which makes the learning much more sample-efficient. Experiments in simulation demonstrate that our approach successfully solves a long-horizon task and is significantly faster than Proximal Policy Optimization (PPO) and the task schema methods.