Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discovery

TL;DR

This paper introduces SD-CQL, a novel offline multi-task reinforcement learning algorithm that discovers skills in a latent space, enabling efficient generalization across multiple tasks with limited data, demonstrated by superior results in StarCraft II.

Contribution

The paper proposes SD-CQL, a task-efficient multi-task offline MARL method that discovers skills in latent space and improves generalization without retraining for new tasks.

Findings

Achieves top performance on 13 out of 14 StarCraft II task sets.

Up to 68.9% improvement on individual tasks.

Demonstrates strong multi-task generalization from limited data.

Abstract

As a data-driven approach, offline MARL learns superior policies solely from offline datasets, ideal for domains rich in historical data but with high interaction costs and risks. However, most existing methods are task-specific, requiring retraining for new tasks, leading to redundancy and inefficiency. To address this issue, we propose a task-efficient value-based multi-task offline MARL algorithm, Skill-Discovery Conservative Q-Learning (SD-CQL). Unlike existing methods decoding actions from skills via behavior cloning, SD-CQL discovers skills in a latent space by reconstructing the next observation, evaluates fixed and variable actions separately, and uses conservative Q-learning with local value calibration to select the optimal action for each skill. It eliminates the need for local-global alignment and enables strong multi-task generalization from limited, small-scale source tasks. Substantial experiments on StarCraft II demonstrate the superior generalization performance and task-efficiency of SD-CQL. It achieves the best performance on $\textbf{13}$ out of $14$ task sets, with up to $\textbf{68.9%}$ improvement on individual task sets.