Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning

TL;DR

This paper introduces ACC-MARL, a framework for multi-agent reinforcement learning that uses automata to efficiently learn and coordinate complex multi-task behaviors in cooperative settings.

Contribution

It presents a novel automata-conditioned approach for multi-agent RL that improves sample efficiency and enables task decomposition and optimal task assignment.

Findings

Demonstrates emergent multi-step coordination among agents.

Shows improved sample efficiency over existing methods.

Validates the approach on complex cooperative tasks.

Abstract

We study the problem of learning multi-task, multi-agent policies for cooperative, temporal objectives, under centralized training, decentralized execution. In this setting, using automata to represent tasks enables the decomposition of complex tasks into simpler sub-tasks that can be assigned to agents. However, existing approaches remain sample-inefficient and are limited to the single-task case. In this work, we present Automata-Conditioned Cooperative Multi-Agent Reinforcement Learning (ACC-MARL), a framework for learning task-conditioned, decentralized team policies. We identify the main challenges to ACC-MARL's feasibility in practice, propose solutions, and prove the correctness of our approach. We further show that the value functions of learned policies can be used to assign tasks optimally at test time. Experiments show emergent task-aware, multi-step coordination among agents, e.g., pressing a button to unlock a door, holding the door, and short-circuiting tasks.