Hybrid Training for Enhanced Multi-task Generalization in Multi-agent Reinforcement Learning

TL;DR

HyGen is a hybrid MARL framework that combines online and offline learning to improve multi-task generalization and training efficiency, demonstrating superior performance on the StarCraft multi-agent challenge.

Contribution

This paper introduces HyGen, a novel hybrid training framework that effectively integrates offline and online learning for enhanced multi-task generalization in MARL.

Findings

Outperforms existing online and offline methods on StarCraft challenge

Effectively extracts and refines general skills from offline datasets

Achieves better generalization to unseen tasks

Abstract

In multi-agent reinforcement learning (MARL), achieving multi-task generalization to diverse agents and objectives presents significant challenges. Existing online MARL algorithms primarily focus on single-task performance, but their lack of multi-task generalization capabilities typically results in substantial computational waste and limited real-life applicability. Meanwhile, existing offline multi-task MARL approaches are heavily dependent on data quality, often resulting in poor performance on unseen tasks. In this paper, we introduce HyGen, a novel hybrid MARL framework, Hybrid Training for Enhanced Multi-Task Generalization, which integrates online and offline learning to ensure both multi-task generalization and training efficiency. Specifically, our framework extracts potential general skills from offline multi-task datasets. We then train policies to select the optimal skills under the centralized training and decentralized execution paradigm (CTDE). During this stage, we utilize a replay buffer that integrates both offline data and online interactions. We empirically demonstrate that our framework effectively extracts and refines general skills, yielding impressive generalization to unseen tasks. Comparative analyses on the StarCraft multi-agent challenge show that HyGen outperforms a wide range of existing solely online and offline methods.