Reinforcement Learning with Parameterized Actions

TL;DR

This paper presents a model-free reinforcement learning algorithm for Markov decision processes with parameterized actions, enabling agents to select actions and their continuous parameters simultaneously, with proven convergence to local optima.

Contribution

The paper introduces the Q-PAMDP algorithm for learning in parameterized action spaces, demonstrating convergence and comparing its performance to direct policy search.

Findings

Q-PAMDP converges to a local optimum.

Q-PAMDP outperforms direct policy search in tested domains.

The approach effectively handles combined discrete and continuous action spaces.

Abstract

We introduce a model-free algorithm for learning in Markov decision processes with parameterized actions-discrete actions with continuous parameters. At each step the agent must select both which action to use and which parameters to use with that action. We introduce the Q-PAMDP algorithm for learning in these domains, show that it converges to a local optimum, and compare it to direct policy search in the goal-scoring and Platform domains.