Conditional Diffusion Model for Multi-Agent Dynamic Task Decomposition

TL;DR

This paper introduces C3DT, a hierarchical multi-agent reinforcement learning framework that uses a conditional diffusion model to automatically infer and coordinate subtasks, improving efficiency and performance in complex dynamic environments.

Contribution

The paper proposes a novel two-level hierarchical MARL framework with a diffusion model for dynamic task decomposition, enabling automatic subtask inference and better coordination.

Findings

C3DT outperforms existing methods on benchmark tasks.

The diffusion-based approach improves sample efficiency.

Enhanced value decomposition through semantic subtask representations.

Abstract

Task decomposition has shown promise in complex cooperative multi-agent reinforcement learning (MARL) tasks, which enables efficient hierarchical learning for long-horizon tasks in dynamic and uncertain environments. However, learning dynamic task decomposition from scratch generally requires a large number of training samples, especially exploring the large joint action space under partial observability. In this paper, we present the Conditional Diffusion Model for Dynamic Task Decomposition (C$\text{D}^\text{3}$T), a novel two-level hierarchical MARL framework designed to automatically infer subtask and coordination patterns. The high-level policy learns subtask representation to generate a subtask selection strategy based on subtask effects. To capture the effects of subtasks on the environment, C$\text{D}^\text{3}$T predicts the next observation and reward using a conditional diffusion model. At the low level, agents collaboratively learn and share specialized skills within their assigned subtasks. Moreover, the learned subtask representation is also used as additional semantic information in a multi-head attention mixing network to enhance value decomposition and provide an efficient reasoning bridge between individual and joint value functions. Experimental results on various benchmarks demonstrate that C$\text{D}^\text{3}$T achieves better performance than existing baselines.