Deep Multi-Agent Reinforcement Learning with Hybrid Action Spaces based on Maximum Entropy

TL;DR

This paper introduces MAHSAC, a novel deep multi-agent reinforcement learning algorithm designed for environments with hybrid action spaces, demonstrating superior performance and stability in complex multi-agent tasks.

Contribution

The paper presents MAHSAC, extending Soft Actor-Critic to handle hybrid action spaces in multi-agent settings under the CTDE paradigm, filling a significant research gap.

Findings

MAHSAC shows fast training and high stability.

It outperforms existing methods in cooperative and competitive scenarios.

Demonstrates effectiveness in environments with hybrid action spaces.

Abstract

Multi-agent deep reinforcement learning has been applied to address a variety of complex problems with either discrete or continuous action spaces and achieved great success. However, most real-world environments cannot be described by only discrete action spaces or only continuous action spaces. And there are few works having ever utilized deep reinforcement learning (drl) to multi-agent problems with hybrid action spaces. Therefore, we propose a novel algorithm: Deep Multi-Agent Hybrid Soft Actor-Critic (MAHSAC) to fill this gap. This algorithm follows the centralized training but decentralized execution (CTDE) paradigm, and extend the Soft Actor-Critic algorithm (SAC) to handle hybrid action space problems in Multi-Agent environments based on maximum entropy. Our experiences are running on an easy multi-agent particle world with a continuous observation and discrete action space, along with some basic simulated physics. The experimental results show that MAHSAC has good performance in training speed, stability, and anti-interference ability. At the same time, it outperforms existing independent deep hybrid learning method in cooperative scenarios and competitive scenarios.