Multi-Pass Q-Networks for Deep Reinforcement Learning with Parameterised Action Spaces

TL;DR

This paper introduces MP-DQN, a novel deep reinforcement learning algorithm for parameterised action spaces, addressing theoretical issues in prior methods and demonstrating superior performance across multiple complex domains.

Contribution

Proposes MP-DQN, a new multi-pass approach that improves theoretical soundness and performance over existing P-DQN algorithms for parameterised actions.

Findings

MP-DQN outperforms P-DQN in data efficiency.

MP-DQN achieves higher converged policy performance.

Demonstrated effectiveness on multiple complex domains.

Abstract

Parameterised actions in reinforcement learning are composed of discrete actions with continuous action-parameters. This provides a framework for solving complex domains that require combining high-level actions with flexible control. The recent P-DQN algorithm extends deep Q-networks to learn over such action spaces. However, it treats all action-parameters as a single joint input to the Q-network, invalidating its theoretical foundations. We analyse the issues with this approach and propose a novel method, multi-pass deep Q-networks, or MP-DQN, to address them. We empirically demonstrate that MP-DQN significantly outperforms P-DQN and other previous algorithms in terms of data efficiency and converged policy performance on the Platform, Robot Soccer Goal, and Half Field Offense domains.