Excluding the Irrelevant: Focusing Reinforcement Learning through Continuous Action Masking

TL;DR

This paper introduces three continuous action masking methods in reinforcement learning to focus on relevant actions based on state, improving training efficiency, safety, and performance in control tasks.

Contribution

The paper proposes novel state-dependent action masking techniques for continuous RL, enhancing learning efficiency and safety by restricting actions to relevant subsets.

Findings

Higher final rewards with masking methods

Faster convergence compared to baseline

Effective in safety-critical applications

Abstract

Continuous action spaces in reinforcement learning (RL) are commonly defined as multidimensional intervals. While intervals usually reflect the action boundaries for tasks well, they can be challenging for learning because the typically large global action space leads to frequent exploration of irrelevant actions. Yet, little task knowledge can be sufficient to identify significantly smaller state-specific sets of relevant actions. Focusing learning on these relevant actions can significantly improve training efficiency and effectiveness. In this paper, we propose to focus learning on the set of relevant actions and introduce three continuous action masking methods for exactly mapping the action space to the state-dependent set of relevant actions. Thus, our methods ensure that only relevant actions are executed, enhancing the predictability of the RL agent and enabling its use in safety-critical applications. We further derive the implications of the proposed methods on the policy gradient. Using proximal policy optimization (PPO), we evaluate our methods on four control tasks, where the relevant action set is computed based on the system dynamics and a relevant state set. Our experiments show that the three action masking methods achieve higher final rewards and converge faster than the baseline without action masking.