Emergency action termination for immediate reaction in hierarchical reinforcement learning

TL;DR

This paper introduces a hierarchical reinforcement learning method that continuously verifies and updates higher-level goals to improve responsiveness and efficiency in large dynamical systems.

Contribution

It proposes a novel approach for real-time validation and replacement of higher-level actions in hierarchical RL, enhancing reactivity and adaptability.

Findings

Effective in seven benchmark environments

Improves reactivity in hierarchical RL

Balances fast training with immediate response

Abstract

Hierarchical decomposition of control is unavoidable in large dynamical systems. In reinforcement learning (RL), it is usually solved with subgoals defined at higher policy levels and achieved at lower policy levels. Reaching these goals can take a substantial amount of time, during which it is not verified whether they are still worth pursuing. However, due to the randomness of the environment, these goals may become obsolete. In this paper, we address this gap in the state-of-the-art approaches and propose a method in which the validity of higher-level actions (thus lower-level goals) is constantly verified at the higher level. If the actions, i.e. lower level goals, become inadequate, they are replaced by more appropriate ones. This way we combine the advantages of hierarchical RL, which is fast training, and flat RL, which is immediate reactivity. We study our approach experimentally on seven benchmark environments.