State Action Separable Reinforcement Learning

TL;DR

This paper introduces State Action Separable Reinforcement Learning (sasRL), a new paradigm that decouples actions from value learning, improving efficiency and convergence in complex decision-making tasks.

Contribution

The paper proposes sasRL, a novel RL framework that separates actions from value functions and incorporates a transition model for enhanced learning efficiency.

Findings

sasRL outperforms traditional RL algorithms by up to 75% in gaming scenarios.

Convergence time of sasRL is theoretically analyzed as $O(T^{1/k})$, showing faster convergence under certain conditions.

Decoupling actions from value functions reduces the complexity of the learning process.

Abstract

Reinforcement Learning (RL) based methods have seen their paramount successes in solving serial decision-making and control problems in recent years. For conventional RL formulations, Markov Decision Process (MDP) and state-action-value function are the basis for the problem modeling and policy evaluation. However, several challenging issues still remain. Among most cited issues, the enormity of state/action space is an important factor that causes inefficiency in accurately approximating the state-action-value function. We observe that although actions directly define the agents' behaviors, for many problems the next state after a state transition matters more than the action taken, in determining the return of such a state transition. In this regard, we propose a new learning paradigm, State Action Separable Reinforcement Learning (sasRL), wherein the action space is decoupled from the value function learning process for higher efficiency. Then, a light-weight transition model is learned to assist the agent to determine the action that triggers the associated state transition. In addition, our convergence analysis reveals that under certain conditions, the convergence time of sasRL is $O(T^{1/k})$, where $T$ is the convergence time for updating the value function in the MDP-based formulation and $k$ is a weighting factor. Experiments on several gaming scenarios show that sasRL outperforms state-of-the-art MDP-based RL algorithms by up to $75\%$.