Macro-Action-Based Multi-Agent/Robot Deep Reinforcement Learning under Partial Observability

TL;DR

This paper introduces macro-action-based deep reinforcement learning methods for multi-agent systems operating under partial observability, enabling asynchronous decision-making and improving scalability in complex real-world tasks.

Contribution

It develops value-based and policy gradient RL algorithms for MacDec-POMDPs, allowing asynchronous macro-action decision-making in multi-agent reinforcement learning.

Findings

Algorithms outperform existing methods in large multi-agent problems

Effective in both simulation and real robot experiments

Demonstrates scalability and high-quality solutions with macro-actions

Abstract

The state-of-the-art multi-agent reinforcement learning (MARL) methods have provided promising solutions to a variety of complex problems. Yet, these methods all assume that agents perform synchronized primitive-action executions so that they are not genuinely scalable to long-horizon real-world multi-agent/robot tasks that inherently require agents/robots to asynchronously reason about high-level action selection at varying time durations. The Macro-Action Decentralized Partially Observable Markov Decision Process (MacDec-POMDP) is a general formalization for asynchronous decision-making under uncertainty in fully cooperative multi-agent tasks. In this thesis, we first propose a group of value-based RL approaches for MacDec-POMDPs, where agents are allowed to perform asynchronous learning and decision-making with macro-action-value functions in three paradigms: decentralized learning and control, centralized learning and control, and centralized training for decentralized execution (CTDE). Building on the above work, we formulate a set of macro-action-based policy gradient algorithms under the three training paradigms, where agents are allowed to directly optimize their parameterized policies in an asynchronous manner. We evaluate our methods both in simulation and on real robots over a variety of realistic domains. Empirical results demonstrate the superiority of our approaches in large multi-agent problems and validate the effectiveness of our algorithms for learning high-quality and asynchronous solutions with macro-actions.