Learning Complex Teamwork Tasks Using a Given Sub-task Decomposition

TL;DR

This paper introduces a sub-task decomposition approach for multi-agent reinforcement learning that improves training efficiency on complex tasks by leveraging expert-provided task breakdowns and fine-tuning sub-teams.

Contribution

It proposes a scalable method that uses sub-task decomposition and policy fine-tuning, addressing issues in naive implementations and enhancing multi-agent learning efficiency.

Findings

Reduces training time for complex tasks compared to from-scratch training.

Addresses key problems in naive sub-task decomposition methods.

Demonstrates empirical benefits across diverse multi-agent tasks.

Abstract

Training a team to complete a complex task via multi-agent reinforcement learning can be difficult due to challenges such as policy search in a large joint policy space, and non-stationarity caused by mutually adapting agents. To facilitate efficient learning of complex multi-agent tasks, we propose an approach which uses an expert-provided decomposition of a task into simpler multi-agent sub-tasks. In each sub-task, a subset of the entire team is trained to acquire sub-task-specific policies. The sub-teams are then merged and transferred to the target task, where their policies are collectively fine-tuned to solve the more complex target task. We show empirically that such approaches can greatly reduce the number of timesteps required to solve a complex target task relative to training from-scratch. However, we also identify and investigate two problems with naive implementations of approaches based on sub-task decomposition, and propose a simple and scalable method to address these problems which augments existing actor-critic algorithms. We demonstrate the empirical benefits of our proposed method, enabling sub-task decomposition approaches to be deployed in diverse multi-agent tasks.