Exploiting Hybrid Policy in Reinforcement Learning for Interpretable Temporal Logic Manipulation

TL;DR

This paper introduces HyTL, a hierarchical reinforcement learning framework guided by linear temporal logic, which enhances exploration efficiency and interpretability in complex robot manipulation tasks.

Contribution

The paper proposes a novel three-level decision framework that integrates temporal logic for improved efficiency and interpretability in reinforcement learning for manipulation tasks.

Findings

HyTL outperforms baseline methods in four manipulation tasks.

The approach improves exploration efficiency and task success rates.

The use of LTL encoding enhances interpretability of policies.

Abstract

Reinforcement Learning (RL) based methods have been increasingly explored for robot learning. However, RL based methods often suffer from low sampling efficiency in the exploration phase, especially for long-horizon manipulation tasks, and generally neglect the semantic information from the task level, resulted in a delayed convergence or even tasks failure. To tackle these challenges, we propose a Temporal-Logic-guided Hybrid policy framework (HyTL) which leverages three-level decision layers to improve the agent's performance. Specifically, the task specifications are encoded via linear temporal logic (LTL) to improve performance and offer interpretability. And a waypoints planning module is designed with the feedback from the LTL-encoded task level as a high-level policy to improve the exploration efficiency. The middle-level policy selects which behavior primitives to execute, and the low-level policy specifies the corresponding parameters to interact with the environment. We evaluate HyTL on four challenging manipulation tasks, which demonstrate its effectiveness and interpretability. Our project is available at: https://sites.google.com/view/hytl-0257/.