Reinforcement Learning With Sparse-Executing Actions via Sparsity Regularization

TL;DR

This paper introduces a new RL framework and algorithm for decision-making tasks where actions can only be executed a limited number of times, addressing a common challenge in real-world applications.

Contribution

The paper formalizes the Sparse Action Markov Decision Process and proposes the ASRE algorithm, which adaptively manages action sparsity during policy learning.

Findings

ASRE effectively handles sparse actions in RL tasks.

ASRE outperforms classical RL algorithms in sparse-action environments.

ASRE improves performance in Atari games with action constraints.

Abstract

Reinforcement learning (RL) has demonstrated impressive performance in decision-making tasks like embodied control, autonomous driving and financial trading. In many decision-making tasks, the agents often encounter the problem of executing actions under limited budgets. However, classic RL methods typically overlook the challenges posed by such sparse-executing actions. They operate under the assumption that all actions can be taken for a unlimited number of times, both in the formulation of the problem and in the development of effective algorithms. To tackle the issue of limited action execution in RL, this paper first formalizes the problem as a Sparse Action Markov Decision Process (SA-MDP), in which specific actions in the action space can only be executed for a limited time. Then, we propose a policy optimization algorithm, Action Sparsity REgularization (ASRE), which adaptively handles each action with a distinct preference. ASRE operates through two steps: First, ASRE evaluates action sparsity by constrained action sampling. Following this, ASRE incorporates the sparsity evaluation into policy learning by way of an action distribution regularization. We provide theoretical identification that validates the convergence of ASRE to a regularized optimal value function. Experiments on tasks with known sparse-executing actions, where classical RL algorithms struggle to train policy efficiently, ASRE effectively constrains the action sampling and outperforms baselines. Moreover, we present that ASRE can generally improve the performance in Atari games, demonstrating its broad applicability.