Exploiting Semantic Epsilon Greedy Exploration Strategy in Multi-Agent Reinforcement Learning

TL;DR

This paper introduces QMIX(SEG), a novel MARL approach that enhances exploration by semantically grouping actions in a hierarchical epsilon greedy strategy, significantly improving performance on complex benchmarks.

Contribution

It proposes a new semantic epsilon greedy exploration strategy integrated with QMIX, enabling more effective exploration in multi-agent reinforcement learning.

Findings

QMIX(SEG) outperforms standard QMIX on SMAC benchmark.

Semantic grouping improves exploration efficiency.

Approach is competitive with state-of-the-art MARL methods.

Abstract

Multi-agent reinforcement learning (MARL) can model many real world applications. However, many MARL approaches rely on epsilon greedy for exploration, which may discourage visiting advantageous states in hard scenarios. In this paper, we propose a new approach QMIX(SEG) for tackling MARL. It makes use of the value function factorization method QMIX to train per-agent policies and a novel Semantic Epsilon Greedy (SEG) exploration strategy. SEG is a simple extension to the conventional epsilon greedy exploration strategy, yet it is experimentally shown to greatly improve the performance of MARL. We first cluster actions into groups of actions with similar effects and then use the groups in a bi-level epsilon greedy exploration hierarchy for action selection. We argue that SEG facilitates semantic exploration by exploring in the space of groups of actions, which have richer semantic meanings than atomic actions. Experiments show that QMIX(SEG) largely outperforms QMIX and leads to strong performance competitive with current state-of-the-art MARL approaches on the StarCraft Multi-Agent Challenge (SMAC) benchmark.