Discrete linear-complexity reinforcement learning in continuous action spaces for Q-learning algorithms

TL;DR

This paper introduces a linear-complexity discretization method for extending Q-learning algorithms to continuous action spaces, enabling efficient reinforcement learning without exponential complexity growth.

Contribution

The paper proposes a novel discretization approach that maintains linear complexity in continuous action space Q-learning, applicable to both shallow and deep neural networks.

Findings

The method achieves linear complexity in discretized action spaces.

It is effective with both shallow and deep neural network architectures.

The approach avoids exponential growth in problem dimensionality.

Abstract

In this article, we sketch an algorithm that extends the Q-learning algorithms to the continuous action space domain. Our method is based on the discretization of the action space. Despite the commonly used discretization methods, our method does not increase the discretized problem dimensionality exponentially. We will show that our proposed method is linear in complexity when the discretization is employed. The variant of the Q-learning algorithm presented in this work, labeled as Finite Step Q-Learning (FSQ), can be deployed to both shallow and deep neural network architectures.