Learning Symbolic Task Decompositions for Multi-Agent Teams

TL;DR

This paper introduces a framework that learns optimal task decompositions for multi-agent systems directly from environment interactions, improving sample efficiency and enabling complex synchronized multi-agent learning without manual design.

Contribution

It presents a novel method that automatically learns symbolic task decompositions and agent policies simultaneously, removing the need for manual task design in multi-agent reinforcement learning.

Findings

Successfully learns task decompositions in various environments

Achieves improved sample efficiency over baseline methods

Enables synchronized multi-agent learning in complex settings

Abstract

One approach for improving sample efficiency in cooperative multi-agent learning is to decompose overall tasks into sub-tasks that can be assigned to individual agents. We study this problem in the context of reward machines: symbolic tasks that can be formally decomposed into sub-tasks. In order to handle settings without a priori knowledge of the environment, we introduce a framework that can learn the optimal decomposition from model-free interactions with the environment. Our method uses a task-conditioned architecture to simultaneously learn an optimal decomposition and the corresponding agents' policies for each sub-task. In doing so, we remove the need for a human to manually design the optimal decomposition while maintaining the sample-efficiency benefits of improved credit assignment. We provide experimental results in several deep reinforcement learning settings, demonstrating the efficacy of our approach. Our results indicate that our approach succeeds even in environments with codependent agent dynamics, enabling synchronous multi-agent learning not achievable in previous works.