Learning to Interrupt: A Hierarchical Deep Reinforcement Learning Framework for Efficient Exploration

TL;DR

This paper introduces a hierarchical deep reinforcement learning framework called Option-Interruption, which integrates human knowledge and an interruption mechanism to improve training efficiency and environmental responsiveness in robotic exploration tasks.

Contribution

It presents a novel hybrid structure combining human-designed options with learnable interruption functions within a hierarchical RL framework, enhancing adaptability and training speed.

Findings

Demonstrates improved training efficiency in navigation tasks

Shows increased flexibility in responding to environmental changes

Validates effectiveness through experiments in four-room navigation and exploration tasks

Abstract

To achieve scenario intelligence, humans must transfer knowledge to robots by developing goal-oriented algorithms, which are sometimes insensitive to dynamically changing environments. While deep reinforcement learning achieves significant success recently, it is still extremely difficult to be deployed in real robots directly. In this paper, we propose a hybrid structure named Option-Interruption in which human knowledge is embedded into a hierarchical reinforcement learning framework. Our architecture has two key components: options, represented by existing human-designed methods, can significantly speed up the training process and interruption mechanism, based on learnable termination functions, enables our system to quickly respond to the external environment. To implement this architecture, we derive a set of update rules based on policy gradient methods and present a complete training process. In the experiment part, our method is evaluated in Four-room navigation and exploration task, which shows the efficiency and flexibility of our framework.